> ## Documentation Index
> Fetch the complete documentation index at: https://gligabao.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Running on Modal

> The ai_train code layout, the modal commands, the sidecar settings and the local tests.

`sidecar/ai_train/`

The training code lives in `sidecar/ai_train/` and runs on [Modal](https://modal.com), the one provider implemented so far. It is ported from `gloflow_phoenix/gf_phoenix/ai_train` and restructured so the cloud-specific parts sit behind a [provider interface](https://gligabao.mintlify.site/training/providers.md). It is the backend behind the [Python sidecar](https://gligabao.mintlify.site/operations/sidecar.md): the sidecar's `modal` backend proxies the Go server's requests to the web API of this app, and nothing else calls it. It runs on Modal and is not imported by `main_sidecar.py`; `Dockerfile.sidecar` copies the directory only so that the sidecar can deploy it. Deploying, stopping and inspecting the app can be done from the sidecar and the admin UI, without the `modal` CLI: see [Modal app lifecycle](https://gligabao.mintlify.site/training/modal.md).

## Layout

```
sidecar/ai_train/
  app.py                      # declares the remote functions + CLI entrypoint (provider-neutral)
  config.py                   # TrainConfig, overridable by env vars
  training_data.py            # the local datasets (build/datasets_training/*/dataset.toml), for the CLI
  requirements_train.txt      # training image (Unsloth stack; 4B, 12B and 31B)
  requirements_infer.txt      # inference-only image (transformers + peft + bitsandbytes; no Unsloth)
  requirements_web.txt        # web API image
  docs/                       # MODEL_SIZES, TRAINING_DATA_FORMAT, tokenizers_and_processors
  test_ai_train_core.py       # local unit tests (no GPU / Modal needed)
  ai_train_core/              # runs inside the container; never imports a provider SDK
    model_config.py           # 4b / 12b / 31b settings (pure data)
    paths.py                  # container mount points + file locations
    prompt_format.py          # Gemma turn-marker format shared by training and inference
    dataset.py                # store / sync / validate datasets on the data volume (internal)
    training_common.py        # dataset loading, SFTConfig, adapter saving
    progress.py               # Trainer callback writing live progress events
    progress_events.py        # the events of a job: write helpers, stop flag (no heavy imports)
    event_log.py              # the job's event log on the provider's key-value store (Modal Dict)
    model_load.py, model_train.py            # training: 4B, 12B and 31B via Unsloth
    model_load_infer.py       # inference loading: base model 4-bit + peft adapter, no Unsloth
    inference.py              # generate with a trained adapter (model_path picks it)
    trained_model.py          # where a job's weights are saved, how they are described, model_path validation
    bootstrap.py              # ping + train dispatch by backend
  api/
    api.py                    # FastAPI app (training jobs, progress, generate, eval; no dataset endpoints)
  providers/
    base.py                   # Provider (ABC), FunctionSpec, ImageSpec
    modal_provider.py         # Modal implementation; the only file that imports modal
    __init__.py               # get_provider(name)
```

## Prerequisites

The `modal` CLI, logged in (`pip install modal && modal setup`), for the commands below (to deploy, stop and look at the app you can use the sidecar and the admin UI instead, see [Modal app lifecycle](https://gligabao.mintlify.site/training/modal.md)), and a provider secret holding `HF_TOKEN` for the Gemma weights (named by `SIDECAR_HF_SECRET`).

## Commands

```sh theme={null}
cd sidecar/ai_train
modal run app.py                                   # ping: image, volumes, GPU info, whether HF_TOKEN is set
modal run app.py --action datasets                 # list the local datasets (build/datasets_training/<name>/dataset.toml)
modal run app.py --action sync --dataset <name>    # upload a local dataset to the data volume (only if it changed)
modal run app.py --action validate --dataset <name>  # check the dataset on the data volume
modal run app.py --action train --dataset <name> --train-steps 100 --model-size 4b   # 4b on A10G, 12b on A100, 31b on A100 80GB
modal run app.py --action infer --model-size 4b --prompt "..."     # run the fine-tuned model (the legacy adapter)
modal run app.py --action infer --model-path trained/<job id> --prompt "..."   # run a trained model
modal serve app.py                                 # web API in dev mode (`modal deploy app.py` to persist)
```

`train` syncs the (local) dataset first; `--dataset` is required for `sync`, `validate` and `train` (there is no default dataset; see [Training datasets](https://gligabao.mintlify.site/training/datasets.md#choosing-the-dataset-of-a-job)). `modal deploy app.py` (or the **Deploy** button of the admin UI's training view, which runs the same deploy through the sidecar: [Modal app lifecycle](https://gligabao.mintlify.site/training/modal.md#deploy)) publishes the [web API](https://gligabao.mintlify.site/training/sidecar-api.md) as the Modal app `gligabao-train`, which the sidecar looks up by name (`SIDECAR_APP_NAME`) and forwards to; the Go server starts jobs through the sidecar (see [Training jobs](https://gligabao.mintlify.site/training/jobs.md)). `modal serve app.py` is the dev mode: it prints a temporary URL that you give the sidecar as `SIDECAR_MODAL_WEB_URL`.

The `modal` CLI uses your own Modal login; the sidecar, which looks the app up through the Modal SDK, needs `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET` instead (see the [sidecar's configuration](https://gligabao.mintlify.site/operations/sidecar.md#configuration)).

Local tests: `cd sidecar/ai_train && python -m unittest test_ai_train_core -v`.

## Configuration

These are the settings of the Modal app, in `config.py`; they can be overridden by environment variable when the app is deployed. The sidecar's own settings (`SIDECAR_BACKEND`, `SIDECAR_MODAL_WEB_URL`, the tokens) are on the [sidecar page](https://gligabao.mintlify.site/operations/sidecar.md#configuration). `SIDECAR_APP_NAME` is read by both and must have the same value.

| Variable | Default | Meaning |
| - | - | - |
| `SIDECAR_PROVIDER` | `modal` | Compute provider (only `modal` exists). |
| `SIDECAR_APP_NAME` | `gligabao-train` | Name of the provider app. |
| `SIDECAR_GPU` | *(empty)* | GPU override for train/generate of every model size (Modal names, e.g. `a100`). Empty means the per-size default: `a10g` for 4b, `a100` for 12b, `a100-80gb` for 31b. |
| `SIDECAR_TRAIN_TIMEOUT_S` | `7200` | Timeout of the train jobs. |
| `SIDECAR_INFER_IDLE_TIMEOUT_S` | `60` | How long a GPU inference container, with its model loaded, stays up after its last request (Modal: 2 to 1200). Requests within that time skip the model load. Redeploy after changing it. See [Warm inference containers](https://gligabao.mintlify.site/training/modal.md#warm-inference-containers). |
| `SIDECAR_HF_SECRET` | *(empty)* | Name of a provider secret with `HF_TOKEN`. Empty means no secret is attached. |
| `SIDECAR_VOL_DATA` | `gligabao-train-data` | Volume mounted at `/data` (datasets). |
| `SIDECAR_VOL_MODELS` | `gligabao-train-models` | Volume mounted at `/models` (the trained models: one directory `trained/<job_id>` per completed job; see [Trained models](https://gligabao.mintlify.site/training/trained-models.md#where-the-weights-are-saved)). |
| `SIDECAR_VOL_EVAL` | `gligabao-train-eval` | Volume mounted at `/eval` (eval examples from the web API). |
| `SIDECAR_VOL_HF_CACHE` | `gligabao-train-hf-cache` | Volume mounted at `/root/.cache/huggingface` (base-model cache, avoids re-downloads). |

Volumes are created if missing.

The Go server's only setting is `sidecar_url`, the address of the sidecar, not of Modal; see the [overview](https://gligabao.mintlify.site/training/overview.md#configuration).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.