Training datasets

internal/api/training/manifest.go, internal/api/training/datasets.go, internal/db/relational/model/training_dataset.go, sidecar/ai_train/api/models.py, sidecar/ai_train/ai_train_core/dataset.py, sidecar/ai_train/ai_train_core/prompt_format.py, sidecar/ai_train/training_data.py, build/datasets_training/ A training dataset is a .jsonl file in a subdirectory of the training datasets directory, described by a dataset.toml manifest. Datasets are managed by the Go server alone: it reads the manifests, keeps the datasets’ records and serves the REST endpoints that external clients and the admin UI use. The sidecar and the Modal app have no dataset endpoints; they only store a dataset internally, on the data volume (/data), when a job needs it. A training dataset is separate from a memory dataset, which is a container of artifacts, entities and edges in the database.

Format

Each line is one training example:

Key	Meaning
`instruction`	The task or user query, e.g. “Find news about NVIDIA and buy 10 shares”.
`input`	Extra context. Often an empty string.
`output`	The expected multi-step answer, including tool usage.

{"instruction": "Check recent news about NVIDIA and buy 10 shares if it's positive", "input": "", "output": "I'll research NVIDIA news... [tool calls and reasoning]"}

Training turns each line into a string with Gemma 4’s turn markers (format_instruction in ai_train_core/prompt_format.py; inference uses the same format):

<|turn>user
{instruction}

{input}<turn|>
<|turn>model
{output}<turn|>

Without an input the blank line and {input} are left out.

<|turn> and <turn|> are single special tokens of Gemma 4 (ids 105 and 106). <turn|> ends a turn and is the token generation stops on (the base model’s generation_config.eos_token_id is [1, 106, 50]). The dataset file has no EOS and should not: format_instruction puts <turn|> at the end of every example, and the tests check against the real Gemma 4 tokenizer that the encoded text ends with id 106. SFTTrainer also appends the tokenizer’s eos_token (<eos>) when the text does not end with it; both stop generation.
Not <start_of_turn> / <end_of_turn>: those are Gemma 2/3’s markers and are not in Gemma 4’s vocabulary, so they are tokenized as plain text. A model trained on them learns to spell <end_of_turn> instead of emitting the stop token and its replies run to max_tokens, inventing further turns. gligabao used them up to 0.46.0; adapters trained with that version must be retrained.
<bos>: Gemma 4’s chat template starts with it, but the tokenizer does not necessarily add it. bos_prefix(tokenizer) prepends it to the training text and to the inference prompt only when the tokenizer does not, so both have exactly one.

load_training_dataset() (training_common.py) reads the file, maps each entry through the formatter into a text field, and builds a Hugging Face Dataset; run_finetuning adds the <bos> prefix and hands it to the SFTTrainer with formatting_func returning that text. The bundled dataset, world-knowledge-base (500 examples of a hand-curated knowledge base of countries, companies, institutions, treaties and sovereign wealth funds), is generated by build/datasets_training/world-knowledge-base/generate_dataset.py.

The training datasets directory and manifests

A training dataset lives in a subdirectory of the training datasets directory and is described by a manifest, a dataset.toml in that subdirectory. gligabao scans the directory for manifests, reads each one to learn the dataset’s name and where its data file is, and offers the datasets for training. The generator script of a dataset and its output sit in the dataset’s own subdirectory.

build/datasets_training/                         the training datasets directory
  world-knowledge-base/               one subdirectory per dataset
    dataset.toml                      the manifest: its presence makes the subdirectory a dataset
    generate_dataset.py               the script that generates the data (ignored by gligabao)
    world_knowledge_base_v1.jsonl     the data file the manifest names
    world_knowledge_base_triples.json raw sources (ignored)
  my-other-dataset/
    dataset.toml
    data/rows.jsonl

The directory is serve --training-dataset-dir <path> or ServeOptions.TrainingDatasetDir when embedding; by default a datasets_training directory next to the database (build/datasets_training for build/gligabao.db; /data/datasets_training in the Docker image). In the repository build/datasets_training/ is tracked while the rest of build/ is ignored, and the generated data files are ignored by a .gitignore in each dataset’s subdirectory: run the generator to create them.

Discovery: the server looks at the direct subdirectories of the training datasets directory and reads the dataset.toml of each (a subdirectory without one is not a dataset; names starting with . are skipped). It does so every time it lists the datasets or starts a job, so a dataset added to the directory, or a regenerated data file, is available at once, with no restart. A directory that does not exist yet has no datasets.
The manifest decides where the data is: gligabao does not search for *.jsonl files, it reads the path in the manifest’s file. Other files in the subdirectory are not part of the dataset. Only the JSONL format is supported for now.

The manifest (`dataset.toml`)

name = "world-knowledge-base"
description = "Memorization of a hand-curated knowledge base ..."
format = "jsonl"
file = "world_knowledge_base_v1.jsonl"
version = "1"
generator = "generate_dataset.py"

Key		Meaning
`file`	required	The data file: a relative path inside the dataset’s subdirectory (no `..`, not absolute), ending in `.jsonl`.
`name`	optional	The dataset’s name, what a training job asks for: 1-122 characters of letters, digits, `.`, `_`, `-`, starting with a letter or digit, without `..`. Default: the subdirectory’s name. Names are unique.
`description`	optional	Free text, shown in the admin UI (the tooltip of the dataset’s row).
`format`	optional	The data format; only `jsonl` (the default).
`version`	optional	Informational.
`generator`	optional	Informational: the script in the subdirectory that creates `file`. The message for a missing `file` names it.

The data file is valid if it is UTF-8, has at least one record, is at most 64 MB, and every non-blank line is a JSON object with string instruction and output and, if present, a string input. A subdirectory with a manifest that cannot be used (a TOML error, no file, a bad name, an unsupported format, a file outside the subdirectory, a missing or invalid data file, a name another dataset already has) is not a dataset: it is skipped, and listed with its message under problems in GET /v1/gb/training/datasets (the admin UI shows them under the datasets table). A job that asks for it gets a 400, like for any unknown dataset.

Names

A dataset has a name (the manifest’s name) and a file name on the sidecar, <name>.jsonl: the file the sidecar stores on its data volume, the filename of the dataset’s record, and the dataset the sidecar reports for a job. A job asks for a dataset by its name.

Adding a dataset

Make build/datasets_training/<dir>/ and put (or generate) the JSONL file in it.
Write build/datasets_training/<dir>/dataset.toml with at least file.
It appears in GET /v1/gb/training/datasets and in the admin UI’s training view; pick it there or give its name as dataset to POST /v1/gb/training/jobs.

Endpoints (Go server)

The Go server reads the manifests, validates, records and lists the datasets; the sidecar is not involved.

Endpoint	Behaviour
`GET /v1/gb/training/datasets[?db=]`	`{"datasets": [...], "problems": [...]}`. `datasets` are the usable datasets by name: the record (below) plus the manifest’s `name`, `description`, `format`, `file`, `version`, `generator`, and `dir` (the subdirectory’s name). `problems` are the subdirectories with an unusable manifest: `{"dir", "error"}`. It never asks the sidecar, so it works with the sidecar or Modal down.
`GET /v1/gb/training/datasets/preview?name=<name>[&lines=][&db=]`	The first lines (default 100, at most 1000) of the dataset’s data file: `{"name", "lines", "num_records", "truncated"}`. `404` for an unknown dataset.
`POST /v1/gb/training/datasets[?db=]`	Uploads a dataset: a multipart form with the file in the field `file` (and optionally `filename`, which overrides the file’s name). The server validates it (as above; a plain `*.jsonl` name) and saves it as the dataset named like the file without `.jsonl`: in the subdirectory of that name, as `<name>.jsonl`, next to a manifest it writes if there is none. Uploading a name again replaces the file. `201` with the dataset, `400` for an invalid file, `409` if a dataset of that name exists with a manifest of its own that names another data file (or the name belongs to a dataset in another subdirectory), `413` if too large.

Only the main database has a training datasets directory; the other databases (the eval one) list no datasets. The upload saves the file first as a temporary file in the dataset’s subdirectory, records it, then renames it into place, so a failure leaves neither a record without its file nor a half-written file.

The `gb_training_datasets` record

One row per dataset (Go: relational.TrainingDataset). The directory is the source of truth: the row is made when the server first sees the data file, and replaced, with status stored again, when it sees the file changed.

Field	Meaning
`id`	The record.
`filename`	`<name>.jsonl`, unique. The dataset’s file name on the sidecar.
`size_bytes`, `num_records`, `sha256`	Size, number of non-blank lines, and hex SHA-256 of the data file.
`status`	`stored` (only on the Go server) or `synced` (a job has taken this version of the file to the sidecar).
`created_at`, `updated_at`	When the record was made; when it last described a new version of the file.
`synced_at`	When the sidecar last took the current file; null while it has not.

Rows of datasets that are no longer in the directory stay in the table and are not listed.

How a dataset reaches the sidecar

Only through a job start. When POST /v1/gb/training/jobs names a dataset, the Go server finds it by name among the manifests (an unknown name is a 400), reads its data file, validates it, brings the record up to date and sends the file to the sidecar in the POST /training/job_start request, as dataset_content (base64) next to dataset (<name>.jsonl). The Go client has no other dataset call. Once the sidecar has taken the job the record becomes synced. The file is sent with every job that uses the dataset, so a sidecar whose volume lost the file still works. A client of the Go server cannot send dataset_content itself. The sidecar (the Modal web API) stores dataset_content on the data volume as /data/<dataset>, replacing an earlier file of that name, and commits the volume before it spawns the job (store_dataset in ai_train_core/dataset.py). 400 if there is no dataset (a job needs one; there is no default dataset), if dataset is not a plain *.jsonl name (no path, no leading dot), if dataset_content is not valid base64 or is empty, or if there is a dataset_content without a dataset. Without dataset_content, dataset must already be on the volume (also 400).

`TrainingDataset` (Python)

TrainingDataset (pydantic, sidecar/ai_train/api/models.py) is how the sidecar loads and passes a dataset around internally; it is not an endpoint body:

Field	Meaning
`filename`	The file name. Required.
`content`	Base64 of the jsonl file. It is never serialized.
`size_bytes`	Size of the file.
`uploaded_at`	When it was stored.
`modified_at`	The file’s modification time.

payload() returns the decoded bytes and raises ValueError if content is missing, empty or not base64. TrainingDataset.from_path(path, with_content=False) loads a file (with its content only if asked). Timestamps are timezone-aware UTC.

Choosing the dataset of a job

A job names its dataset by name, in the required dataset field of POST /v1/gb/training/jobs: one of the datasets listed by GET /v1/gb/training/datasets (the training view of the admin UI fills its picker from it). The modal run app.py CLI reads the same manifests (training_data.py, from build/datasets_training; set GLIGABAO_DATASETS_TRAINING_DIR to use another directory; it needs Python 3.11+ for tomllib):

modal run app.py --action datasets                          # list the local datasets and their problems
modal run app.py --action sync --dataset <name>             # upload the dataset to the data volume, as <name>.jsonl, if it changed
modal run app.py --action validate --dataset <name>         # check the file on the data volume
modal run app.py --action train --dataset <name> --train-steps 100 --model-size 4b   # syncs, then trains

--dataset is required for sync, validate and train. A dataset is also what a trained model can be evaluated on: the Audit of a model asks it the questions of the dataset it was trained on (a training data audit), so the dataset must stay in this directory.

Getting started

Concepts

Guides

Evals

Training

Training datasets

Format

The training datasets directory and manifests

The manifest (`dataset.toml`)

Names

Adding a dataset

Endpoints (Go server)

The `gb_training_datasets` record

How a dataset reaches the sidecar

`TrainingDataset` (Python)

Choosing the dataset of a job

​Format

​The training datasets directory and manifests

​The manifest (dataset.toml)

​Names

​Adding a dataset

​Endpoints (Go server)

​The gb_training_datasets record

​How a dataset reaches the sidecar

​TrainingDataset (Python)

​Choosing the dataset of a job

Format

The training datasets directory and manifests

The manifest (`dataset.toml`)

Names

Adding a dataset

Endpoints (Go server)

The `gb_training_datasets` record

How a dataset reaches the sidecar

`TrainingDataset` (Python)

Choosing the dataset of a job