Training jobs

internal/sidecar/training_jobs.go, internal/sidecar/training_events.go, internal/api/training/jobs.go, internal/api/training/monitor.go, internal/db/relational/model/training_job_run.go, sidecar/ai_train/api/models.py, admin_ui/src/training.ts How the Go server starts a training job through the sidecar, follows its progress and keeps track of it. The Go server talks only to the sidecar; the sidecar proxies each call to the Modal web API, so every sidecar answer below is Modal’s, passed on unchanged. The overview shows the flow.

The `TrainingJob` struct

The sidecar reports a started job as a TrainingJob; the Go client decodes the same fields into its own struct.

Field	Type	Meaning
`job_id`	string	The sidecar’s ID of the job; it names the job’s event log. Never empty: the Go client returns an error if the sidecar sends none.
`provider_job_id`	string	The compute provider’s ID of the spawned call (for the provider’s dashboard). Optional.
`status`	string	What the sidecar reports when the job starts: `started`.
`model_size`	string	`4b` or `12b`.
`num_train_steps`	integer	How many steps the job was asked to train.
`dataset`	string	The file on the data volume the job trains on; empty for the default dataset.
`started_at`	RFC 3339 timestamp	When the sidecar started the job. Always timezone-aware UTC, so the Go side can parse it.

The two definitions:

Python, sidecar/ai_train/api/models.py: the pydantic model TrainingJob. job_id has a minimum length of 1, num_train_steps must be greater than 0, and a naive started_at is treated as local time and converted to UTC.
Go, internal/sidecar/training_jobs.go: sidecar.TrainingJob, with the JSON tags above (StartedAt is a time.Time).

Example answer of POST /training/job_start:

{
  "job_id": "3f9a1c52d0e84b17",
  "provider_job_id": "fc-01JABCDEF",
  "status": "started",
  "model_size": "12b",
  "num_train_steps": 500,
  "dataset": "my.jsonl",
  "started_at": "2026-10-26T09:30:12.481530Z"
}

Starting a job

POST /v1/gb/training/jobs[?db=<name>], part of the API (served whether or not the admin UI is on; the paths were /v1/gb/admin/api/training/... before gligabao 0.30.0). db picks the database the job is recorded in (the first one, standard, without it). The body is optional JSON; every field is optional too:

Field	Default	Meaning
`num_train_steps`	`100`	Training steps. Must be positive.
`model_size`	`4b`	`4b` or `12b` (case-insensitive). See Model sizes.
`dataset`	the sidecar’s default dataset	File name of an uploaded training dataset (a `*.jsonl`, no path), as listed by `GET /v1/gb/training/datasets`. The server sends its file to the sidecar with the job. See Training datasets.

An empty body starts a 4b job of 100 steps on the default dataset. A field left out is not sent, so the defaults are the sidecar’s (sidecar.DefaultNumTrainSteps, sidecar.DefaultModelSize in Go).

curl -X POST http://localhost:7878/v1/gb/training/jobs \
  -H "Content-Type: application/json" \
  -d '{"num_train_steps": 500, "model_size": "12b", "dataset": "my.jsonl"}'

The answer is 201 with the stored row:

{
  "id": 3,
  "job_id": "3f9a1c52d0e84b17",
  "provider_job_id": "fc-01JABCDEF",
  "status": "started",
  "model_size": "12b",
  "dataset": "my.jsonl",
  "num_train_steps": 500,
  "last_step": 0,
  "last_loss": null,
  "final_loss": null,
  "started_at": "2026-10-26T09:30:12Z",
  "created_at": "2026-10-26T09:30:13Z",
  "finished_at": null
}

Status	When
`201`	The job was started and recorded.
`400`	The body is not valid JSON, `num_train_steps` is negative, `model_size` is not `4b` or `12b`, or `dataset` is not a plain `*.jsonl` file name (the sidecar is not called). Also for a `dataset` the server has no record of (upload it first), and when the sidecar itself rejects the request as invalid: the answer then carries the sidecar’s message.
`404`	`db` is not one of the configured databases.
`502`	The sidecar could not be reached, or it answered with another error: its own `502` when it cannot reach Modal (the app is not deployed, or its URL cannot be found), or an error status from the Modal web API. Nothing is recorded.
`503`	The server has no sidecar client: the sidecar failed its health check when the server started (“sidecar is not active”). Start the sidecar and restart the server.
`500`	The dataset’s file is missing from the dataset directory or does not match its record (upload it again); or the job was started, but the row could not be written or read back. The message names the `job_id`, so the job can still be found on the provider.

Every job the sidecar reports as started is a new row, even if an identical one was started a minute before. A start that fails records nothing. The row is written even if the HTTP client disconnects while the job is being started, because by then the job is already running. Once it is written, the training monitor starts following the job.

Listing jobs and datasets

Endpoint	Answer
`POST /v1/gb/training/jobs/{id}/stop[?db=<name>]`	Asks the sidecar to stop the job run `{id}` (the Stop job button of the admin UI). `202 {"id", "status": "stopping"}`; the job ends at its next step with a `cancelled` event and the row follows like for any other end. `404` for an unknown run, `409` for one that is over already, `502` if the sidecar fails, `503` without a sidecar.
`GET /v1/gb/training/jobs[?db=<name>][&limit=<n>]`	`{"jobs": [...]}`: the rows of `gb_training_job_runs`, newest first. `limit` defaults to 50, at most 500 (`400` if it is not a positive integer).
`GET /v1/gb/training/models[?db=<name>][&limit=<n>]`	`{"models": [...]}`: the rows of `gb_trained_models`, newest first (default 100, at most 500). See Trained models.
`GET /v1/gb/training/models/{id}[?db=<name>]`	One trained model; `404` if unknown.
`POST /v1/gb/training/models/{id}/infer[?db=<name>]`	Body `{"prompt", "max_tokens"?, "temperature"?}`: runs the prompt through that model on the sidecar and answers `{"model_id", "response"}`. See Trained models.
`GET /v1/gb/training/datasets[?db=<name>]`	`{"datasets": [...]}`: the training datasets of the database (`filename`, `size_bytes`, `num_records`, `sha256`, `status`, `updated_at`, `synced_at`), latest first. Read from the database only: no sidecar needed. Uploading is `POST /v1/gb/training/datasets`; see Training datasets.

Training progress

A job reports as it runs. The training container writes events (status changes, and the step and loss of every trainer log) and the sidecar streams them at GET /training/job_events. The training monitor of the Go server (internal/api/training/monitor.go) follows that stream for every job started through the start endpoint:

it stores each event in gb_training_job_events and updates the job’s row: status follows the status events, last_step and last_loss the progress events, and the end sets final_loss (completed), error (failed, cancelled) and finished_at;
it forwards each event to the SSE feed;
when the sidecar ends the stream without the job being over (the sidecar limits one response to a few minutes) or the connection breaks, it reconnects from the next event it has not stored, waiting longer between attempts (1 second, doubling up to 30) while nothing arrives;
when the server starts, it resumes following every job that is not completed, failed or cancelled, from the last stored event;
if the sidecar reports no end within 3 hours of the job’s start (the provider cancels a job after SIDECAR_TRAIN_TIMEOUT_S, 2 hours by default, and a container that dies writes nothing), it marks the job failed with an explanation and a final event.

The SSE feed

GET /v1/gb/training/events[?db=<name>][&run_id=<id>] is a Server-Sent Events stream. Each message has event: status or event: progress and a data: JSON object with the sidecar’s event fields plus db, run_id, job_id and seq (the event’s index in the job):

id: 4
event: progress
data: {"db":"standard","run_id":3,"job_id":"3f9a1c52d0e84b17","seq":4,"type":"progress","step":20,"loss":1.234,"learning_rate":0.0002,"epoch":0.3,"elapsed_seconds":42.1,"timestamp":1792169586.9}

With run_id: the stream of that job. It first replays the events already stored (all of them, or those after the Last-Event-ID that a reconnecting EventSource sends, or from from=<n>), then sends the live ones. Each message has id: <seq>. The stream stays open after the job’s last event; the client closes it. 404 for an unknown run.
Without run_id: only the live events of every job of the database, no history, no ids. The admin UI uses it to keep the job list current.

A client that cannot keep up (more than 256 events behind) is disconnected; it reconnects and is caught up from the database. A : keepalive comment is sent every 15 seconds.

The training view

The training view of the admin UI puts this together. The Run button posts to the start endpoint with the chosen model size, dataset (picked from the uploaded datasets) and steps. The table lists the jobs (status, model, dataset, last_step / num_train_steps, loss, start time) and is updated from the feed without polling. Selecting a job shows its event feed and a graph of its loss against the step, built from the stored events and then the live ones. The view also uploads datasets (a file input and a table of the uploaded ones with their status). It opens its two SSE connections only while it is shown.

The `gb_training_job_runs` table

One row per training job started. Created with the rest of the schema, on both SQLite and PostgreSQL.

Column	Meaning
`id`	The record.
`job_id`	The sidecar’s job ID (names the event log there). Indexed, not unique.
`provider_job_id`	The compute provider’s ID of the call. Empty if the sidecar did not send one.
`status`	`started` when the sidecar reports the job started, then the job’s status events: `running`, `training`, and finally `completed`, `failed` or `cancelled` (stopped on request).
`model_size`	`4b` or `12b`.
`dataset`	The dataset file the job trains on; empty for the sidecar’s default dataset.
`num_train_steps`	Steps the job was asked to train.
`last_step`, `last_loss`	From the newest progress event (`last_loss` is null until a progress event has a loss).
`final_loss`	The training loss reported when the job completed. Null before.
`error`	Why the job failed. Empty otherwise.
`started_at`	When the sidecar says the job started.
`created_at`	When the row was made.
`finished_at`	When the job completed, failed or was cancelled. Null while it runs.

Trained models

A trained model is a set of weights: for the LoRA fine-tuning the sidecar does, the few files of an adapter. Every job that completes makes a new trained model, a row of the gb_trained_models table, so you can train many models of the same type with other data, steps or size, and each stays available to run inference on. A cancelled or failed job saves nothing. The job saves its adapter in /models/trained/<job_id> of the models volume and reports it in its completed event; the monitor records it. Where the files are, how the row is made, how inference picks a model and how to add an inference endpoint are on Trained models. The training view of the admin UI lists the models; clicking one shows its settings and files and a form that runs a prompt through it.

Column of `gb_trained_models`	Meaning
`id`	The record.
`name`	For lists; made from the size, dataset, steps and the start of the job id (`4b-mine.jsonl-100steps-1a2b3c4d`).
`run_id`	The job run (`gb_training_job_runs.id`) that made it; null if that run was deleted.
`job_id`	The sidecar’s job ID. Unique: a job makes one model.
`base_model`, `model_size`	The model that was fine-tuned (e.g. `google/gemma-4-E4B-it`) and its size, `4b` or `12b`. Inference needs the size to pick the hardware.
`dataset`, `dataset_sha256`, `num_examples`	The dataset file name (empty for the sidecar’s default), the digest of the file the job read (the file of that name can be uploaded again with other content) and its number of examples.
`num_train_steps`, `final_loss`	How long the job trained and its mean training loss.
`params`	JSON object: the other training settings.
`path`	The directory of the weights on the models volume, relative to it (`trained/<job id>`): the `model_path` of inference.
`files`, `size_bytes`	JSON array of `{path, size_bytes, sha256}` for the files in that directory, and their total size.
`created_at`	When the row was made.

From Go (relational package): RecordTrainedModel(ctx, db, model) stores a TrainedModel and returns its ID and whether it is new (a job that already has a model keeps it); GetTrainedModel(ctx, db, id) reads one, or returns relational.ErrNotFound; ListTrainedModels(ctx, db, limit) lists them, newest first. The Go client has Infer(ctx, sidecar.InferRequest).

The `gb_training_job_events` table

One row per event the sidecar reported for a job, so a job’s feed and loss graph can be shown after it finished.

Column	Meaning
`run_id`	The job run (`gb_training_job_runs.id`); the rows are deleted with it.
`seq`	The event’s index in the sidecar’s event log. `(run_id, seq)` is the primary key, so an event the stream replays is stored once, and a reconnect resumes after the highest `seq`.
`type`	`status` or `progress`.
`data`	The event as JSON.
`created_at`	When the row was made.

Timestamps follow the usual convention of the backend (see Database): RFC 3339 UTC text with second precision on SQLite, TIMESTAMPTZ on PostgreSQL. From Go (relational package): RecordTrainingJobRun(ctx, db, run) stores a TrainingJobRun and returns its ID (id, created_at and the fields that only events set are ignored); GetTrainingJobRun(ctx, db, id) reads one, or returns relational.ErrNotFound; ListTrainingJobRuns(ctx, db, limit, unfinished) lists them (newest first, or only the unfinished ones, oldest first); UpdateTrainingJobRun(ctx, db, id, update) writes the fields that are set in a TrainingJobRunUpdate; RecordTrainingJobEvent, ListTrainingJobEvents and NextTrainingJobEventSeq handle the events. A job that outlives its timeout (SIDECAR_TRAIN_TIMEOUT_S, default 7200 seconds) is cancelled by the provider. If its container could not report that, the monitor marks the job failed after 3 hours (see Training progress). Database files created by an earlier version are not migrated: delete the SQLite file (or drop gb_training_job_runs on PostgreSQL) and the server recreates the tables.

The Go client

internal/sidecar is the only package that knows the sidecar’s HTTP API (see The Go client).

c := sidecar.New("")   // "" is sidecar.DefaultBaseURL

job, err := c.StartTrainingJob(ctx, sidecar.StartTrainingJobRequest{
    NumTrainSteps: 500,                  // 0 = the sidecar's default (100)
    ModelSize:     sidecar.ModelSize12B, // "" = the sidecar's default (4b)
    Dataset:       "my.jsonl",           // "" = the sidecar's default dataset
    // DatasetContent: the dataset's jsonl file, base64; the sidecar stores it as Dataset before it starts the job
})
// job.JobID, job.ProviderJobID, job.Status, job.ModelSize, job.Dataset, job.NumTrainSteps, job.StartedAt

// Follow a job: fn gets each event from index `from` on; nil is returned when the sidecar ends the stream.
err = c.StreamTrainingJobEvents(ctx, job.JobID, 0, func(ev sidecar.TrainingEvent) error {
    // ev.Seq, ev.Type ("status" or "progress"), ev.Status, ev.Step, ev.Loss, ev.FinalLoss, ev.Error, ...
    return nil // returning an error stops the stream
})

Infer(ctx, sidecar.InferRequest{Prompt, ModelSize, ModelPath, ...}) calls POST /infer/query and waits as long as the context allows (a cold start takes minutes); a generation that fails on the sidecar is a *sidecar.InferError. CancelTrainingJob(ctx, jobID) calls POST /training/job_cancel (a *sidecar.StatusError with 404 or 409 for an unknown or finished job). StartTrainingJob returns an error for a negative NumTrainSteps, for any status other than 200 from the sidecar (a *sidecar.StatusError), for a body that is not the expected JSON, and for an answer without a job_id. StreamTrainingJobEvents returns nil when the sidecar ends the stream (after the job’s last event, or after the sidecar’s time limit; reconnect with from = the last Seq + 1), fn’s error if fn fails, the context’s error when it is done, and an error for a failed connection, a bad status or an event that is not JSON. The optional fields of sidecar.TrainingEvent are pointers, nil when the event does not have them. There are no dataset calls: the sidecar has no dataset endpoints, and a dataset travels in DatasetContent (see Training datasets). The tests (internal/sidecar/*_test.go, internal/api/training/*_test.go) run against an httptest server and a fake sidecar, so they need neither Python nor a cloud account.

Getting started

Concepts

Guides

Evals

Training

The `TrainingJob` struct

Starting a job

Listing jobs and datasets

Training progress

The SSE feed

The training view

The `gb_training_job_runs` table

Trained models

The `gb_training_job_events` table

The Go client

​The TrainingJob struct

​Starting a job

​Listing jobs and datasets

​Training progress

​The SSE feed

​The training view

​The gb_training_job_runs table

​Trained models

​The gb_training_job_events table

​The Go client

The `TrainingJob` struct

Starting a job

Listing jobs and datasets

Training progress

The SSE feed

The training view

The `gb_training_job_runs` table

Trained models

The `gb_training_job_events` table

The Go client