> ## Documentation Index > Fetch the complete documentation index at: https://gligabao.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Training datasets > The dataset file format, the training datasets directory and its dataset.toml manifests, how the Go server lists training datasets, and how a dataset reaches the sidecar inside a job start. `internal/api/training/manifest.go`, `internal/api/training/datasets.go`, `internal/db/relational/model/training_dataset.go`, `sidecar/ai_train/api/models.py`, `sidecar/ai_train/ai_train_core/dataset.py`, `sidecar/ai_train/ai_train_core/prompt_format.py`, `sidecar/ai_train/training_data.py`, `build/datasets_training/` A training dataset is a `.jsonl` file in a subdirectory of the training datasets directory, described by a `dataset.toml` manifest. **Datasets are managed by the Go server alone**: it reads the manifests, keeps the datasets' records and serves the REST endpoints that external clients and the admin UI use. The sidecar and the Modal app have no dataset endpoints; they only store a dataset internally, on the data volume (`/data`), when a job needs it. A training dataset is separate from a memory [dataset](https://gligabao.mintlify.site/concepts/datasets.md), which is a container of artifacts, entities and edges in the database. ## Format Each line is one training example: | Key | Meaning | | - | - | | `instruction` | The task or user query, e.g. "Find news about NVIDIA and buy 10 shares". | | `input` | Extra context. Often an empty string. | | `output` | The expected multi-step answer, including tool usage. | ```json theme={null} {"instruction": "Check recent news about NVIDIA and buy 10 shares if it's positive", "input": "", "output": "I'll research NVIDIA news... [tool calls and reasoning]"} ``` Training turns each line into a string with Gemma 4's turn markers (`format_instruction` in `ai_train_core/prompt_format.py`; inference uses the same format): ``` <|turn>user {instruction} {input} <|turn>model {output} ``` Without an `input` the blank line and `{input}` are left out. * **`<|turn>` and ``** are single special tokens of Gemma 4 (ids 105 and 106). `` ends a turn and is the token generation stops on (the base model's `generation_config.eos_token_id` is `[1, 106, 50]`). The dataset file has no EOS and should not: `format_instruction` puts `` at the end of every example, and the tests check against the real Gemma 4 tokenizer that the encoded text ends with id 106. `SFTTrainer` also appends the tokenizer's `eos_token` (``) when the text does not end with it; both stop generation. * **Not `` / ``:** those are Gemma 2/3's markers and are not in Gemma 4's vocabulary, so they are tokenized as plain text. A model trained on them learns to spell `` instead of emitting the stop token and its replies run to `max_tokens`, inventing further turns. gligabao used them up to 0.46.0; **adapters trained with that version must be retrained.** * **``:** Gemma 4's chat template starts with it, but the tokenizer does not necessarily add it. `bos_prefix(tokenizer)` prepends it to the training text and to the inference prompt only when the tokenizer does not, so both have exactly one. `load_training_dataset()` (`training_common.py`) reads the file, maps each entry through the formatter into a `text` field, and builds a Hugging Face `Dataset`; `run_finetuning` adds the `` prefix and hands it to the `SFTTrainer` with `formatting_func` returning that `text`. The bundled dataset, `world-knowledge-base` (500 examples of a hand-curated knowledge base of countries, companies, institutions, treaties and sovereign wealth funds), is generated by `build/datasets_training/world-knowledge-base/generate_dataset.py`. ## The training datasets directory and manifests A training dataset lives in a subdirectory of the **training datasets directory** and is described by a **manifest**, a `dataset.toml` in that subdirectory. gligabao scans the directory for manifests, reads each one to learn the dataset's name and where its data file is, and offers the datasets for training. The generator script of a dataset and its output sit in the dataset's own subdirectory. ``` build/datasets_training/ the training datasets directory world-knowledge-base/ one subdirectory per dataset dataset.toml the manifest: its presence makes the subdirectory a dataset generate_dataset.py the script that generates the data (ignored by gligabao) world_knowledge_base_v1.jsonl the data file the manifest names world_knowledge_base_triples.json raw sources (ignored) my-other-dataset/ dataset.toml data/rows.jsonl ``` The directory is `serve --training-dataset-dir ` or `ServeOptions.TrainingDatasetDir` when embedding; by default a `datasets_training` directory next to the database (`build/datasets_training` for `build/gligabao.db`; `/data/datasets_training` in the Docker image). In the repository `build/datasets_training/` is tracked while the rest of `build/` is ignored, and the generated data files are ignored by a `.gitignore` in each dataset's subdirectory: run the generator to create them. * **Discovery:** the server looks at the direct subdirectories of the training datasets directory and reads the `dataset.toml` of each (a subdirectory without one is not a dataset; names starting with `.` are skipped). It does so every time it lists the datasets or starts a job, so a dataset added to the directory, or a regenerated data file, is available at once, with no restart. A directory that does not exist yet has no datasets. * **The manifest decides where the data is:** gligabao does not search for `*.jsonl` files, it reads the path in the manifest's `file`. Other files in the subdirectory are not part of the dataset. Only the JSONL format is supported for now. ### The manifest (`dataset.toml`) ```toml theme={null} name = "world-knowledge-base" description = "Memorization of a hand-curated knowledge base ..." format = "jsonl" file = "world_knowledge_base_v1.jsonl" version = "1" generator = "generate_dataset.py" ``` | Key | | Meaning | | - | - | - | | `file` | required | The data file: a relative path inside the dataset's subdirectory (no `..`, not absolute), ending in `.jsonl`. | | `name` | optional | The dataset's name, what a training job asks for: 1-122 characters of letters, digits, `.`, `_`, `-`, starting with a letter or digit, without `..`. Default: the subdirectory's name. Names are unique. | | `description` | optional | Free text, shown in the admin UI (the tooltip of the dataset's row). | | `format` | optional | The data format; only `jsonl` (the default). | | `version` | optional | Informational. | | `generator` | optional | Informational: the script in the subdirectory that creates `file`. The message for a missing `file` names it. | The data file is valid if it is UTF-8, has at least one record, is at most 64 MB, and every non-blank line is a JSON object with string `instruction` and `output` and, if present, a string `input`. A subdirectory with a manifest that cannot be used (a TOML error, no `file`, a bad `name`, an unsupported `format`, a `file` outside the subdirectory, a missing or invalid data file, a name another dataset already has) is not a dataset: it is skipped, and listed with its message under `problems` in `GET /v1/gb/training/datasets` (the admin UI shows them under the datasets table). A job that asks for it gets a `400`, like for any unknown dataset. ### Names A dataset has a **name** (the manifest's `name`) and a **file name on the sidecar**, `.jsonl`: the file the sidecar stores on its data volume, the `filename` of the dataset's record, and the `dataset` the sidecar reports for a job. A job asks for a dataset by its name. ### Adding a dataset 1. Make `build/datasets_training//` and put (or generate) the JSONL file in it. 2. Write `build/datasets_training//dataset.toml` with at least `file`. 3. It appears in `GET /v1/gb/training/datasets` and in the admin UI's training view; pick it there or give its name as `dataset` to `POST /v1/gb/training/jobs`. ## Endpoints (Go server) The Go server reads the manifests, validates, records and lists the datasets; the sidecar is not involved. | Endpoint | Behaviour | | - | - | | `GET /v1/gb/training/datasets[?db=]` | `{"datasets": [...], "problems": [...]}`. `datasets` are the usable datasets by name: the record (below) plus the manifest's `name`, `description`, `format`, `file`, `version`, `generator`, and `dir` (the subdirectory's name). `problems` are the subdirectories with an unusable manifest: `{"dir", "error"}`. It never asks the sidecar, so it works with the sidecar or Modal down. | | `GET /v1/gb/training/datasets/preview?name=[&lines=][&db=]` | The first lines (default 100, at most 1000) of the dataset's data file: `{"name", "lines", "num_records", "truncated"}`. `404` for an unknown dataset. | | `POST /v1/gb/training/datasets[?db=]` | Uploads a dataset: a multipart form with the file in the field `file` (and optionally `filename`, which overrides the file's name). The server validates it (as above; a plain `*.jsonl` name) and saves it as the dataset named like the file without `.jsonl`: in the subdirectory of that name, as `.jsonl`, next to a manifest it writes if there is none. Uploading a name again replaces the file. `201` with the dataset, `400` for an invalid file, `409` if a dataset of that name exists with a manifest of its own that names another data file (or the name belongs to a dataset in another subdirectory), `413` if too large. | Only the main database has a training datasets directory; the other databases (the eval one) list no datasets. The upload saves the file first as a temporary file in the dataset's subdirectory, records it, then renames it into place, so a failure leaves neither a record without its file nor a half-written file. ### The `gb_training_datasets` record One row per dataset (Go: `relational.TrainingDataset`). The directory is the source of truth: the row is made when the server first sees the data file, and replaced, with status `stored` again, when it sees the file changed. | Field | Meaning | | - | - | | `id` | The record. | | `filename` | `.jsonl`, unique. The dataset's file name on the sidecar. | | `size_bytes`, `num_records`, `sha256` | Size, number of non-blank lines, and hex SHA-256 of the data file. | | `status` | `stored` (only on the Go server) or `synced` (a job has taken this version of the file to the sidecar). | | `created_at`, `updated_at` | When the record was made; when it last described a new version of the file. | | `synced_at` | When the sidecar last took the current file; null while it has not. | Rows of datasets that are no longer in the directory stay in the table and are not listed. ## How a dataset reaches the sidecar Only through a job start. When `POST /v1/gb/training/jobs` names a `dataset`, the Go server finds it by name among the manifests (an unknown name is a `400`), reads its data file, validates it, brings the record up to date and sends the file to the sidecar in the `POST /training/job_start` request, as `dataset_content` (base64) next to `dataset` (`.jsonl`). The Go client has no other dataset call. Once the sidecar has taken the job the record becomes `synced`. The file is sent with every job that uses the dataset, so a sidecar whose volume lost the file still works. A client of the Go server cannot send `dataset_content` itself. The sidecar (the Modal web API) stores `dataset_content` on the data volume as `/data/`, replacing an earlier file of that name, and commits the volume before it spawns the job (`store_dataset` in `ai_train_core/dataset.py`). `400` if there is no `dataset` (a job needs one; there is no default dataset), if `dataset` is not a plain `*.jsonl` name (no path, no leading dot), if `dataset_content` is not valid base64 or is empty, or if there is a `dataset_content` without a `dataset`. Without `dataset_content`, `dataset` must already be on the volume (also `400`). ## `TrainingDataset` (Python) `TrainingDataset` (pydantic, `sidecar/ai_train/api/models.py`) is how the sidecar loads and passes a dataset around internally; it is not an endpoint body: | Field | Meaning | | - | - | | `filename` | The file name. Required. | | `content` | Base64 of the jsonl file. It is **never serialized**. | | `size_bytes` | Size of the file. | | `uploaded_at` | When it was stored. | | `modified_at` | The file's modification time. | `payload()` returns the decoded bytes and raises `ValueError` if `content` is missing, empty or not base64. `TrainingDataset.from_path(path, with_content=False)` loads a file (with its content only if asked). Timestamps are timezone-aware UTC. ## Choosing the dataset of a job A job names its dataset by name, in the required `dataset` field of `POST /v1/gb/training/jobs`: one of the datasets listed by `GET /v1/gb/training/datasets` (the **training** view of the admin UI fills its picker from it). The `modal run app.py` CLI reads the same manifests (`training_data.py`, from `build/datasets_training`; set `GLIGABAO_DATASETS_TRAINING_DIR` to use another directory; it needs Python 3.11+ for `tomllib`): ```sh theme={null} modal run app.py --action datasets # list the local datasets and their problems modal run app.py --action sync --dataset # upload the dataset to the data volume, as .jsonl, if it changed modal run app.py --action validate --dataset # check the file on the data volume modal run app.py --action train --dataset --train-steps 100 --model-size 4b # syncs, then trains ``` `--dataset` is required for `sync`, `validate` and `train`. A dataset is also what a trained model can be evaluated on: the **Audit** of a model asks it the questions of the dataset it was trained on (a [training data audit](https://gligabao.mintlify.site/training/audit.md)), so the dataset must stay in this directory. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.