Running on Modal

sidecar/ai_train/ The training code lives in sidecar/ai_train/ and runs on Modal, the one provider implemented so far. It is ported from gloflow_phoenix/gf_phoenix/ai_train and restructured so the cloud-specific parts sit behind a provider interface. It is the backend behind the Python sidecar: the sidecar’s modal backend proxies the Go server’s requests to the web API of this app, and nothing else calls it. It runs on Modal and is not imported by main_sidecar.py; Dockerfile.sidecar copies the directory only so that the sidecar can deploy it. Deploying, stopping and inspecting the app can be done from the sidecar and the admin UI, without the modal CLI: see Modal app lifecycle.

Layout

sidecar/ai_train/
  app.py                      # declares the remote functions + CLI entrypoint (provider-neutral)
  config.py                   # TrainConfig, overridable by env vars
  training_data.py            # the local datasets (build/datasets_training/*/dataset.toml), for the CLI
  requirements_train.txt      # training image (Unsloth stack; 4B, 12B and 31B)
  requirements_infer.txt      # inference-only image (transformers + peft + bitsandbytes; no Unsloth)
  requirements_web.txt        # web API image
  docs/                       # MODEL_SIZES, TRAINING_DATA_FORMAT, tokenizers_and_processors
  test_ai_train_core.py       # local unit tests (no GPU / Modal needed)
  ai_train_core/              # runs inside the container; never imports a provider SDK
    model_config.py           # 4b / 12b / 31b settings (pure data)
    paths.py                  # container mount points + file locations
    prompt_format.py          # Gemma turn-marker format shared by training and inference
    dataset.py                # store / sync / validate datasets on the data volume (internal)
    training_common.py        # dataset loading, SFTConfig, adapter saving
    progress.py               # Trainer callback writing live progress events
    progress_events.py        # the events of a job: write helpers, stop flag (no heavy imports)
    event_log.py              # the job's event log on the provider's key-value store (Modal Dict)
    model_load.py, model_train.py            # training: 4B, 12B and 31B via Unsloth
    model_load_infer.py       # inference loading: base model 4-bit + peft adapter, no Unsloth
    inference.py              # generate with a trained adapter (model_path picks it)
    trained_model.py          # where a job's weights are saved, how they are described, model_path validation
    bootstrap.py              # ping + train dispatch by backend
  api/
    api.py                    # FastAPI app (training jobs, progress, generate, eval; no dataset endpoints)
  providers/
    base.py                   # Provider (ABC), FunctionSpec, ImageSpec
    modal_provider.py         # Modal implementation; the only file that imports modal
    __init__.py               # get_provider(name)

Prerequisites

The modal CLI, logged in (pip install modal && modal setup), for the commands below (to deploy, stop and look at the app you can use the sidecar and the admin UI instead, see Modal app lifecycle), and a provider secret holding HF_TOKEN for the Gemma weights (named by SIDECAR_HF_SECRET).

Commands

cd sidecar/ai_train
modal run app.py                                   # ping: image, volumes, GPU info, whether HF_TOKEN is set
modal run app.py --action datasets                 # list the local datasets (build/datasets_training/<name>/dataset.toml)
modal run app.py --action sync --dataset <name>    # upload a local dataset to the data volume (only if it changed)
modal run app.py --action validate --dataset <name>  # check the dataset on the data volume
modal run app.py --action train --dataset <name> --train-steps 100 --model-size 4b   # 4b on A10G, 12b on A100, 31b on A100 80GB
modal run app.py --action infer --model-size 4b --prompt "..."     # run the fine-tuned model (the legacy adapter)
modal run app.py --action infer --model-path trained/<job id> --prompt "..."   # run a trained model
modal serve app.py                                 # web API in dev mode (`modal deploy app.py` to persist)

train syncs the (local) dataset first; --dataset is required for sync, validate and train (there is no default dataset; see Training datasets). modal deploy app.py (or the Deploy button of the admin UI’s training view, which runs the same deploy through the sidecar: Modal app lifecycle) publishes the web API as the Modal app gligabao-train, which the sidecar looks up by name (SIDECAR_APP_NAME) and forwards to; the Go server starts jobs through the sidecar (see Training jobs). modal serve app.py is the dev mode: it prints a temporary URL that you give the sidecar as SIDECAR_MODAL_WEB_URL. The modal CLI uses your own Modal login; the sidecar, which looks the app up through the Modal SDK, needs MODAL_TOKEN_ID and MODAL_TOKEN_SECRET instead (see the sidecar’s configuration). Local tests: cd sidecar/ai_train && python -m unittest test_ai_train_core -v.

Configuration

These are the settings of the Modal app, in config.py; they can be overridden by environment variable when the app is deployed. The sidecar’s own settings (SIDECAR_BACKEND, SIDECAR_MODAL_WEB_URL, the tokens) are on the sidecar page. SIDECAR_APP_NAME is read by both and must have the same value.

Variable	Default	Meaning
`SIDECAR_PROVIDER`	`modal`	Compute provider (only `modal` exists).
`SIDECAR_APP_NAME`	`gligabao-train`	Name of the provider app.
`SIDECAR_GPU`	(empty)	GPU override for train/generate of every model size (Modal names, e.g. `a100`). Empty means the per-size default: `a10g` for 4b, `a100` for 12b, `a100-80gb` for 31b.
`SIDECAR_TRAIN_TIMEOUT_S`	`7200`	Timeout of the train jobs.
`SIDECAR_INFER_IDLE_TIMEOUT_S`	`60`	How long a GPU inference container, with its model loaded, stays up after its last request (Modal: 2 to 1200). Requests within that time skip the model load. Redeploy after changing it. See Warm inference containers.
`SIDECAR_HF_SECRET`	(empty)	Name of a provider secret with `HF_TOKEN`. Empty means no secret is attached.
`SIDECAR_VOL_DATA`	`gligabao-train-data`	Volume mounted at `/data` (datasets).
`SIDECAR_VOL_MODELS`	`gligabao-train-models`	Volume mounted at `/models` (the trained models: one directory `trained/<job_id>` per completed job; see Trained models).
`SIDECAR_VOL_EVAL`	`gligabao-train-eval`	Volume mounted at `/eval` (eval examples from the web API).
`SIDECAR_VOL_HF_CACHE`	`gligabao-train-hf-cache`	Volume mounted at `/root/.cache/huggingface` (base-model cache, avoids re-downloads).

Volumes are created if missing. The Go server’s only setting is sidecar_url, the address of the sidecar, not of Modal; see the overview.

Getting started

Concepts

Guides

Evals

Training

Layout

Prerequisites

Commands

Configuration

​Layout

​Prerequisites

​Commands

​Configuration

Layout

Prerequisites

Commands

Configuration