> ## Documentation Index > Fetch the complete documentation index at: https://gligabao.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Training jobs > The TrainingJob struct, the training endpoints that start and list jobs and stream their progress, the gb_training_job_runs and gb_training_job_events tables and the Go client calls. `internal/sidecar/training_jobs.go`, `internal/sidecar/training_events.go`, `internal/api/training/jobs.go`, `internal/api/training/monitor.go`, `internal/db/relational/model/training_job_run.go`, `sidecar/ai_train/api/models.py`, `admin_ui/src/training.ts` How the Go server starts a training job through the sidecar, follows its progress and keeps track of it. The Go server talks only to the sidecar; the sidecar proxies each call to the Modal web API, so every sidecar answer below is Modal's, passed on unchanged. The [overview](https://gligabao.mintlify.site/training/overview.md#how-a-job-starts) shows the flow. ## The `TrainingJob` struct The sidecar reports a started job as a `TrainingJob`; the Go client decodes the same fields into its own struct. | Field | Type | Meaning | | - | - | - | | `job_id` | string | The sidecar's ID of the job; it names the job's event log. Never empty: the Go client returns an error if the sidecar sends none. | | `provider_job_id` | string | The compute provider's ID of the spawned call (for the provider's dashboard). Optional. | | `status` | string | What the sidecar reports when the job starts: `started`. | | `model_size` | string | `4b` or `12b`. | | `num_train_steps` | integer | How many steps the job was asked to train. | | `dataset` | string | The file on the data volume the job trains on; empty for the default dataset. | | `started_at` | RFC 3339 timestamp | When the sidecar started the job. Always timezone-aware UTC, so the Go side can parse it. | The two definitions: * **Python**, `sidecar/ai_train/api/models.py`: the pydantic model `TrainingJob`. `job_id` has a minimum length of 1, `num_train_steps` must be greater than 0, and a naive `started_at` is treated as local time and converted to UTC. * **Go**, `internal/sidecar/training_jobs.go`: `sidecar.TrainingJob`, with the JSON tags above (`StartedAt` is a `time.Time`). Example answer of `POST /training/job_start`: ```json theme={null} { "job_id": "3f9a1c52d0e84b17", "provider_job_id": "fc-01JABCDEF", "status": "started", "model_size": "12b", "num_train_steps": 500, "dataset": "my.jsonl", "started_at": "2026-10-26T09:30:12.481530Z" } ``` ## Starting a job `POST /v1/gb/training/jobs[?db=]`, part of the API (served whether or not the [admin UI](https://gligabao.mintlify.site/getting-started/configuration.md#admin-ui) is on; the paths were `/v1/gb/admin/api/training/...` before gligabao 0.30.0). `db` picks the database the job is recorded in (the first one, `standard`, without it). The body is optional JSON; every field is optional too: | Field | Default | Meaning | | - | - | - | | `num_train_steps` | `100` | Training steps. Must be positive. | | `model_size` | `4b` | `4b` or `12b` (case-insensitive). See [Model sizes](https://gligabao.mintlify.site/training/models.md). | | `dataset` | the sidecar's default dataset | File name of an uploaded training dataset (a `*.jsonl`, no path), as listed by `GET /v1/gb/training/datasets`. The server sends its file to the sidecar with the job. See [Training datasets](https://gligabao.mintlify.site/training/datasets.md#how-a-dataset-reaches-the-sidecar). | An empty body starts a 4b job of 100 steps on the default dataset. A field left out is not sent, so the defaults are the sidecar's (`sidecar.DefaultNumTrainSteps`, `sidecar.DefaultModelSize` in Go). ```sh theme={null} curl -X POST http://localhost:7878/v1/gb/training/jobs \ -H "Content-Type: application/json" \ -d '{"num_train_steps": 500, "model_size": "12b", "dataset": "my.jsonl"}' ``` The answer is `201` with the stored row: ```json theme={null} { "id": 3, "job_id": "3f9a1c52d0e84b17", "provider_job_id": "fc-01JABCDEF", "status": "started", "model_size": "12b", "dataset": "my.jsonl", "num_train_steps": 500, "last_step": 0, "last_loss": null, "final_loss": null, "started_at": "2026-10-26T09:30:12Z", "created_at": "2026-10-26T09:30:13Z", "finished_at": null } ``` | Status | When | | - | - | | `201` | The job was started and recorded. | | `400` | The body is not valid JSON, `num_train_steps` is negative, `model_size` is not `4b` or `12b`, or `dataset` is not a plain `*.jsonl` file name (the sidecar is not called). Also for a `dataset` the server has no record of (upload it first), and when the sidecar itself rejects the request as invalid: the answer then carries the sidecar's message. | | `404` | `db` is not one of the configured databases. | | `502` | The sidecar could not be reached, or it answered with another error: its own `502` when it cannot reach Modal (the app is not deployed, or its URL cannot be found), or an error status from the Modal web API. Nothing is recorded. | | `503` | The server has no sidecar client: the sidecar failed its health check when the server started ("sidecar is not active"). Start the sidecar and restart the server. | | `500` | The dataset's file is missing from the dataset directory or does not match its record (upload it again); or the job was started, but the row could not be written or read back. The message names the `job_id`, so the job can still be found on the provider. | Every job the sidecar reports as started is a new row, even if an identical one was started a minute before. A start that fails records nothing. The row is written even if the HTTP client disconnects while the job is being started, because by then the job is already running. Once it is written, the [training monitor](#training-progress) starts following the job. ## Listing jobs and datasets | Endpoint | Answer | | - | - | | `POST /v1/gb/training/jobs/{id}/stop[?db=]` | Asks the sidecar to stop the job run `{id}` (the **Stop job** button of the admin UI). `202 {"id", "status": "stopping"}`; the job ends at its next step with a `cancelled` event and the row follows like for any other end. `404` for an unknown run, `409` for one that is over already, `502` if the sidecar fails, `503` without a sidecar. | | `GET /v1/gb/training/jobs[?db=][&limit=]` | `{"jobs": [...]}`: the rows of `gb_training_job_runs`, newest first. `limit` defaults to 50, at most 500 (`400` if it is not a positive integer). | | `GET /v1/gb/training/models[?db=][&limit=]` | `{"models": [...]}`: the rows of `gb_trained_models`, newest first (default 100, at most 500). See [Trained models](https://gligabao.mintlify.site/training/trained-models.md). | | `GET /v1/gb/training/models/{id}[?db=]` | One trained model; `404` if unknown. | | `POST /v1/gb/training/models/{id}/infer[?db=]` | Body `{"prompt", "max_tokens"?, "temperature"?}`: runs the prompt through that model on the sidecar and answers `{"model_id", "response"}`. See [Trained models](https://gligabao.mintlify.site/training/trained-models.md#how-inference-runs). | | `GET /v1/gb/training/datasets[?db=]` | `{"datasets": [...]}`: the training datasets of the database (`filename`, `size_bytes`, `num_records`, `sha256`, `status`, `updated_at`, `synced_at`), latest first. Read from the database only: no sidecar needed. Uploading is `POST /v1/gb/training/datasets`; see [Training datasets](https://gligabao.mintlify.site/training/datasets.md#managing-datasets-go-server). | ## Training progress A job reports as it runs. The training container writes [events](https://gligabao.mintlify.site/training/sidecar-api.md#progress-events) (status changes, and the step and loss of every trainer log) and the sidecar streams them at `GET /training/job_events`. The **training monitor** of the Go server (`internal/api/training/monitor.go`) follows that stream for every job started through the start endpoint: * it stores each event in `gb_training_job_events` and updates the job's row: `status` follows the status events, `last_step` and `last_loss` the progress events, and the end sets `final_loss` (`completed`), `error` (`failed`, `cancelled`) and `finished_at`; * it forwards each event to the SSE feed; * when the sidecar ends the stream without the job being over (the sidecar limits one response to a few minutes) or the connection breaks, it reconnects from the next event it has not stored, waiting longer between attempts (1 second, doubling up to 30) while nothing arrives; * when the server starts, it resumes following every job that is not `completed`, `failed` or `cancelled`, from the last stored event; * if the sidecar reports no end within 3 hours of the job's start (the provider cancels a job after `SIDECAR_TRAIN_TIMEOUT_S`, 2 hours by default, and a container that dies writes nothing), it marks the job `failed` with an explanation and a final event. ### The SSE feed `GET /v1/gb/training/events[?db=][&run_id=]` is a Server-Sent Events stream. Each message has `event: status` or `event: progress` and a `data:` JSON object with the sidecar's event fields plus `db`, `run_id`, `job_id` and `seq` (the event's index in the job): ``` id: 4 event: progress data: {"db":"standard","run_id":3,"job_id":"3f9a1c52d0e84b17","seq":4,"type":"progress","step":20,"loss":1.234,"learning_rate":0.0002,"epoch":0.3,"elapsed_seconds":42.1,"timestamp":1792169586.9} ``` * **With `run_id`**: the stream of that job. It first replays the events already stored (all of them, or those after the `Last-Event-ID` that a reconnecting `EventSource` sends, or from `from=`), then sends the live ones. Each message has `id: `. The stream stays open after the job's last event; the client closes it. `404` for an unknown run. * **Without `run_id`**: only the live events of every job of the database, no history, no ids. The admin UI uses it to keep the job list current. A client that cannot keep up (more than 256 events behind) is disconnected; it reconnects and is caught up from the database. A `: keepalive` comment is sent every 15 seconds. ### The training view The **training** view of the [admin UI](https://gligabao.mintlify.site/getting-started/configuration.md#admin-ui) puts this together. The `Run` button posts to the start endpoint with the chosen model size, dataset (picked from the uploaded datasets) and steps. The table lists the jobs (status, model, dataset, `last_step / num_train_steps`, loss, start time) and is updated from the feed without polling. Selecting a job shows its event feed and a graph of its loss against the step, built from the stored events and then the live ones. The view also uploads datasets (a file input and a table of the uploaded ones with their status). It opens its two SSE connections only while it is shown. ## The `gb_training_job_runs` table One row per training job started. Created with the rest of the schema, on both SQLite and PostgreSQL. | Column | Meaning | | - | - | | `id` | The record. | | `job_id` | The sidecar's job ID (names the event log there). Indexed, not unique. | | `provider_job_id` | The compute provider's ID of the call. Empty if the sidecar did not send one. | | `status` | `started` when the sidecar reports the job started, then the job's status events: `running`, `training`, and finally `completed`, `failed` or `cancelled` (stopped on request). | | `model_size` | `4b` or `12b`. | | `dataset` | The dataset file the job trains on; empty for the sidecar's default dataset. | | `num_train_steps` | Steps the job was asked to train. | | `last_step`, `last_loss` | From the newest progress event (`last_loss` is null until a progress event has a loss). | | `final_loss` | The training loss reported when the job completed. Null before. | | `error` | Why the job failed. Empty otherwise. | | `started_at` | When the sidecar says the job started. | | `created_at` | When the row was made. | | `finished_at` | When the job completed, failed or was cancelled. Null while it runs. | ## Trained models A trained model is a set of weights: for the LoRA fine-tuning the sidecar does, the few files of an adapter. Every job that **completes** makes a **new** trained model, a row of the `gb_trained_models` table, so you can train many models of the same type with other data, steps or size, and each stays available to run inference on. A cancelled or failed job saves nothing. The job saves its adapter in `/models/trained/` of the models volume and reports it in its `completed` event; the monitor records it. Where the files are, how the row is made, how inference picks a model and how to add an inference endpoint are on [Trained models](https://gligabao.mintlify.site/training/trained-models.md). The **training** view of the admin UI lists the models; clicking one shows its settings and files and a form that runs a prompt through it. | Column of `gb_trained_models` | Meaning | | - | - | | `id` | The record. | | `name` | For lists; made from the size, dataset, steps and the start of the job id (`4b-mine.jsonl-100steps-1a2b3c4d`). | | `run_id` | The job run (`gb_training_job_runs.id`) that made it; null if that run was deleted. | | `job_id` | The sidecar's job ID. Unique: a job makes one model. | | `base_model`, `model_size` | The model that was fine-tuned (e.g. `google/gemma-4-E4B-it`) and its size, `4b` or `12b`. Inference needs the size to pick the hardware. | | `dataset`, `dataset_sha256`, `num_examples` | The dataset file name (empty for the sidecar's default), the digest of the file the job read (the file of that name can be uploaded again with other content) and its number of examples. | | `num_train_steps`, `final_loss` | How long the job trained and its mean training loss. | | `params` | JSON object: the other training settings. | | `path` | The directory of the weights on the models volume, relative to it (`trained/`): the `model_path` of inference. | | `files`, `size_bytes` | JSON array of `{path, size_bytes, sha256}` for the files in that directory, and their total size. | | `created_at` | When the row was made. | From Go (`relational` package): `RecordTrainedModel(ctx, db, model)` stores a `TrainedModel` and returns its ID and whether it is new (a job that already has a model keeps it); `GetTrainedModel(ctx, db, id)` reads one, or returns `relational.ErrNotFound`; `ListTrainedModels(ctx, db, limit)` lists them, newest first. The Go client has `Infer(ctx, sidecar.InferRequest)`. ## The `gb_training_job_events` table One row per event the sidecar reported for a job, so a job's feed and loss graph can be shown after it finished. | Column | Meaning | | - | - | | `run_id` | The job run (`gb_training_job_runs.id`); the rows are deleted with it. | | `seq` | The event's index in the sidecar's event log. `(run_id, seq)` is the primary key, so an event the stream replays is stored once, and a reconnect resumes after the highest `seq`. | | `type` | `status` or `progress`. | | `data` | The event as JSON. | | `created_at` | When the row was made. | Timestamps follow the usual convention of the backend (see [Database](https://gligabao.mintlify.site/getting-started/database.md)): RFC 3339 UTC text with second precision on SQLite, `TIMESTAMPTZ` on PostgreSQL. From Go (`relational` package): `RecordTrainingJobRun(ctx, db, run)` stores a `TrainingJobRun` and returns its ID (`id`, `created_at` and the fields that only events set are ignored); `GetTrainingJobRun(ctx, db, id)` reads one, or returns `relational.ErrNotFound`; `ListTrainingJobRuns(ctx, db, limit, unfinished)` lists them (newest first, or only the unfinished ones, oldest first); `UpdateTrainingJobRun(ctx, db, id, update)` writes the fields that are set in a `TrainingJobRunUpdate`; `RecordTrainingJobEvent`, `ListTrainingJobEvents` and `NextTrainingJobEventSeq` handle the events. A job that outlives its timeout (`SIDECAR_TRAIN_TIMEOUT_S`, default 7200 seconds) is cancelled by the provider. If its container could not report that, the monitor marks the job `failed` after 3 hours (see [Training progress](#training-progress)). Database files created by an earlier version are not migrated: delete the SQLite file (or drop `gb_training_job_runs` on PostgreSQL) and the server recreates the tables. ## The Go client `internal/sidecar` is the only package that knows the sidecar's HTTP API (see [The Go client](https://gligabao.mintlify.site/operations/sidecar.md#the-go-client)). ```go theme={null} c := sidecar.New("") // "" is sidecar.DefaultBaseURL job, err := c.StartTrainingJob(ctx, sidecar.StartTrainingJobRequest{ NumTrainSteps: 500, // 0 = the sidecar's default (100) ModelSize: sidecar.ModelSize12B, // "" = the sidecar's default (4b) Dataset: "my.jsonl", // "" = the sidecar's default dataset // DatasetContent: the dataset's jsonl file, base64; the sidecar stores it as Dataset before it starts the job }) // job.JobID, job.ProviderJobID, job.Status, job.ModelSize, job.Dataset, job.NumTrainSteps, job.StartedAt // Follow a job: fn gets each event from index `from` on; nil is returned when the sidecar ends the stream. err = c.StreamTrainingJobEvents(ctx, job.JobID, 0, func(ev sidecar.TrainingEvent) error { // ev.Seq, ev.Type ("status" or "progress"), ev.Status, ev.Step, ev.Loss, ev.FinalLoss, ev.Error, ... return nil // returning an error stops the stream }) ``` `Infer(ctx, sidecar.InferRequest{Prompt, ModelSize, ModelPath, ...})` calls `POST /infer/query` and waits as long as the context allows (a cold start takes minutes); a generation that fails on the sidecar is a `*sidecar.InferError`. `CancelTrainingJob(ctx, jobID)` calls `POST /training/job_cancel` (a `*sidecar.StatusError` with 404 or 409 for an unknown or finished job). `StartTrainingJob` returns an error for a negative `NumTrainSteps`, for any status other than 200 from the sidecar (a `*sidecar.StatusError`), for a body that is not the expected JSON, and for an answer without a `job_id`. `StreamTrainingJobEvents` returns nil when the sidecar ends the stream (after the job's last event, or after the sidecar's time limit; reconnect with `from` = the last `Seq` + 1), `fn`'s error if `fn` fails, the context's error when it is done, and an error for a failed connection, a bad status or an event that is not JSON. The optional fields of `sidecar.TrainingEvent` are pointers, nil when the event does not have them. There are no dataset calls: the sidecar has no dataset endpoints, and a dataset travels in `DatasetContent` (see [Training datasets](https://gligabao.mintlify.site/training/datasets.md#how-a-dataset-reaches-the-sidecar)). The tests (`internal/sidecar/*_test.go`, `internal/api/training/*_test.go`) run against an `httptest` server and a fake sidecar, so they need neither Python nor a cloud account. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.