sidecar/routes.py, sidecar/backends/, sidecar/ai_train/api/api.py, sidecar/ai_train/api/models.py
There are two layers with the same routes. The sidecar (sidecar/routes.py, see the service) is the REST API the Go server calls; it does not implement the routes, it forwards each one unchanged to its backend and streams the answer back. The Modal web API, described on this page, is what the modal backend forwards to: a FastAPI app created by create_api in api/api.py and served by modal deploy app.py (or modal serve app.py in dev, with SIDECAR_MODAL_WEB_URL). It is provider neutral: everything provider-specific (spawning a job, calling a function, committing and reloading volumes) is injected as callables by app.py. Only the sidecar calls it; the Go server never does. The request and answer formats below are the same on both layers, since the sidecar changes nothing.
What the sidecar adds: GET /health (answered by the sidecar itself, not forwarded), the routes that manage the Modal app itself (/backends/...: status, deploy, cancel, stop, volumes; see Modal app lifecycle) and 502 {"error": ...} when the backend cannot be reached (for modal, also when the deployed app’s web URL cannot be found).
There are no dataset endpoints (
/datasets/upload and /datasets/list were removed in gligabao 0.31.0): training datasets belong to the Go server, which hands one to the sidecar inside POST /training/job_start; see Training datasets.
The routes were put under these prefixes in gligabao 0.26.2. They used to be /start_training_job, /get_training_progress, /generate_response, /add_to_eval_dataset, /get_eval_dataset_count, /upload_dataset and /list_datasets.
POST /training/job_start
Body {dataset, num_train_steps?, model_size?, dataset_content?} (defaults 100 and 4b; dataset is required, there is no default dataset). dataset_content is the dataset’s jsonl file, base64; when it is given the sidecar stores it on the data volume as dataset before it spawns the job (the Go server sends it for every job that uses one of its datasets). The sidecar generates the job’s ID, spawns the job and answers with a TrainingJob. 400 with {"error": ...} for a model_size that is not 4b, 12b or 31b, for a num_train_steps that is not a positive integer, for a missing dataset, for a dataset that is not a plain *.jsonl file name, for a dataset_content that is not valid base64, is empty or has no dataset, and for a dataset without dataset_content that is not on the data volume. It returns as soon as the job is spawned.
Progress events
The training run appends one JSON object per event to the job’s event log: entries in a Modal Dict named<SIDECAR_APP_NAME>-events, which drops each entry 7 days after its last use (the Go database is the permanent record). There is no volume, commit or reload. An event is one request to the Dict, tried again if it fails; a progress or log event that still fails is skipped, a status event that fails raises. An event without a type is a progress event.
Status events, {"type": "status", "status": ..., "timestamp": ...}:
completed, failed and cancelled end the job. Log events, {"type": "log", "message": ...}, say what the job is doing: importing the libraries, the dataset loaded, the model loaded, the trainer set up, the adapters saved. Progress events, one per trainer step (logging_steps=1):
POST /training/job_cancel
Body {"job_id": ...}. Asks the job to stop: the web container sets the job’s stop flag in the Dict, and the training container looks at it between steps (at most every 2 seconds) and between the stages of its setup. It then ends the job with a cancelled event, without saving the adapters. So the stop takes effect at the next step, or, during the model load, when the load is done; it is not instant. Answers 200 {"job_id": ..., "status": "stopping"}, 400 for a bad job_id, 404 for an unknown job (one started through job_start is known at once, before its container runs) and 409 for one that is over already. The web container writes no events here.
GET /training/job_events
?job_id=<id>[&from=<n>] (400 without a valid job_id). A Server-Sent Events stream of the job’s events as they are written, from the event with index n (default 0):
id is the event’s index in the event log, event its type. While nothing new arrives, : keepalive comment lines are sent. The web container looks for new events every second. The stream ends after the completed, failed or cancelled event, or after 5 minutes whatever happened; a reader that wants more reconnects with from=<last id + 1>, and gets every event it missed, however long it was away (within the 7 days). If the job has no events yet (it is still waiting for a GPU) the stream just waits.
Watchdog. A container that is killed (out of memory, a timeout) cannot report its end. While a stream is idle, every 15 seconds the web container asks Modal about the job’s call (its id is stored by job_start); if the call is over and the log has no final event, it writes a failed event with Modal’s error, and the stream delivers it. The Go monitor therefore learns of such a failure at once, not after its 3 hour limit. The Go client reads it with StreamTrainingJobEvents, see Training progress.
GET /training/job_progress
?job_id=<id> (400 without a valid one): all the events of that job’s event log so far, as {"progress": [...]} (an empty list when there are none). Until 0.44.0 it also answered without job_id with the newest job. Unlike the stream it answers once with everything so far. There is no Go client method for it; the Go server uses the stream.
POST /infer/query
Body {prompt, max_tokens?, temperature?, model_size?, model_path?} (defaults 1024, 0.7 and 4b): runs a fine-tuned adapter and answers with the result of the inference function. model_path (trained/<job id>, the path of a trained model) picks the weights; model_size must be the size they were trained for. Without model_path the adapter that versions before 0.38.0 saved per size (/models/finetuned_model_<size>) is used. 400 for an invalid model_size or model_path; an adapter that is not on the volume answers {"error": ...}. The models volume is reloaded first so a newly trained adapter is seen. The answer also has cold_start (true when this request had to load the model), load_seconds (how long that took; the container start is not included) and input_tokens / output_tokens: the container keeps the model loaded and stays up for SIDECAR_INFER_IDLE_TIMEOUT_S (default 60 seconds) after its last request, so only the first request pays the load (see Warm inference containers; what a cold request does in the container is in Cold and warm starts).
GET /models/file
?model_path=trained/<job id>&file=<path> answers the raw bytes (application/octet-stream) of one file of a trained model: file is a path of the model’s files (relative to its directory, nested paths allowed). It exists so the Go server can fetch a model’s weights to its own disk for a deployment, since the web container mounts neither the models volume nor the HF cache. model_path must have the shape trained/<job id> and file must be a relative path of plain segments (no .., no absolute path): else 400. A volume or file that does not exist is 404; a backend without such storage is 404 too; a failure reading it is 502 ({"error"}).
The Modal backend (backends/modal_backend.py, model_file) runs python -m providers.modal.read_model_file <model_path> <file> (ai_train/providers/modal/read_model_file.py, read only, like the volume listing) in a subprocess and streams its stdout, so a multi-gigabyte file is never held in memory. The first chunk is read before the answer starts, so a missing file is a clean 404; a failure after that aborts the response, which the Go side sees as a short body. In Go, sidecar.Client.DownloadModelFile(ctx, modelPath, file, w) streams it into a writer, and internal/training.FetchModelFiles downloads every file of a model into a directory through a temporary name, checks the size and the sha256 recorded in gb_trained_models.files, and renames it into place; a file already there with the right size and sha256 is not downloaded again.
Eval examples
POST /evals/add_to_dataset takes {prompt, response, notes?} (400 if the prompt or response is missing), appends it with a timestamp to eval_dataset.jsonl on the eval volume and commits the volume. GET /evals/get_dataset_count answers {"count": n}, the number of lines of that file. These examples are collected for later evaluation; they are not training data.
Tests
sidecar/ai_train/test_ai_train_core.py tests the API with FastAPI’s test client and injected fakes (no Modal, GPU or ML libraries): cd sidecar/ai_train && python -m unittest test_ai_train_core -v.