Skip to main content
sidecar/ai_train/ai_train_core/trained_model.py, model_train.py, inference.py, sidecar/ai_train/api/api.py, sidecar/ai_train/app.py, internal/api/training/models.go, internal/api/training/monitor.go, internal/sys/sidecar/inference.go, internal/db/relational/ A trained model is a set of weights: for the LoRA fine-tuning the sidecar does, the few files of an adapter (adapter_config.json, adapter_model.safetensors, the tokenizer files). Each job that completes makes a new trained model, one row of the gb_trained_models table, so a user can train many models of the same type with other data, steps or size, and every one stays available to run inference on. A cancelled or failed job saves nothing.

​
Where the weights are saved

The weights live on the provider’s models volume (Modal: SIDECAR_VOL_MODELS, default gligabao-train-models, mounted at /models in the containers). Each job gets a directory of its own, named after the job’s ID, and nothing ever writes into another job’s directory:
  • Only the adapter is saved, not the base model. The base model (google/gemma-4-E4B-it, google/gemma-4-12B-it or google/gemma-4-31B-it) is downloaded from Hugging Face into the HF cache volume (SIDECAR_VOL_HF_CACHE, mounted at /root/.cache/huggingface) and shared by all jobs. adapter_config.json names it, and inference loads it (4-bit) and attaches the adapter. A trained model therefore needs the base model to stay downloadable (HF_TOKEN, see SIDECAR_HF_SECRET in Running on Modal) and its model_size to be right. The adapter is small next to the base model (the code’s estimate is 64 MB and up; the 12B one is larger); size_bytes has the real number.
  • Who writes it. Only the training container, as the last stage of a job: it saves the adapter, commits the models volume (commit_fn([MODELS_DIR])), then lists the files and hashes them. Only after that does the job report completed, so a model row never points at weights that were not committed. A job that fails or is cancelled earlier saves nothing; one that dies while saving can leave a partial directory, which has no row.
  • Who reads it. The inference containers mount the models and HF cache volumes (INFER_VOLS in app.py). A container that was already running does not see a volume change committed elsewhere, so generate_response reloads the models volume first (a failing reload is logged and the request goes on with what the container sees); that is how an adapter trained a minute ago is found. The web container mounts neither volume (it only needs data and eval).
  • Never overwritten, never deleted. A job ID is used once, so a path is never written twice. gligabao has no endpoint that deletes weights or a model row; to free space, remove the directory yourself (modal volume rm -r gligabao-train-models trained/<job_id>); the row stays and inference on it then answers 502 with the sidecar’s “No trained model at …” message.
  • Looking at them. The admin UI’s Volumes button (training view, on the app’s row; see Modal app lifecycle) shows the volume’s entry count; modal volume ls gligabao-train-models trained/<job_id> lists a model’s files and modal volume get gligabao-train-models trained/<job_id> ./out downloads them. The row’s files (path, size, sha256 of each) lets you verify a copy with sha256sum.
  • Runs outside the server. modal run app.py --action train also saves to trained/<job_id> (the job ID is a Unix timestamp there), but it bypasses the sidecar and the monitor, so no row is made; such a model can still be used with modal run app.py --action generate --model-path trained/<timestamp>.

​
How a model is recorded

  1. The training container saves the adapter in /models/trained/<job_id> and lists its files with their sizes and sha256 (ai_train_core/trained_model.py).
  2. The job’s completed event carries that as model: {base_model, model_size, path, files: [{path, size_bytes, sha256}], size_bytes, num_examples, dataset_sha256, params}. path is relative to the models volume; dataset_sha256 is the digest of the dataset file the container actually read (the file of that name can be uploaded again with other content); params are the LoRA rank and alpha, dropout, learning rate, batch size, gradient accumulation, sequence length and 4-bit flag.
  3. The Go monitor records it before it stores the event (internal/api/training/monitor.go; the monitor is described in Training progress), so a failure to record makes the event stream retry instead of losing the model. The row (gb_trained_models): run_id (the job’s gb_training_job_runs row; null if that row is deleted later), unique job_id, name (made from size, dataset, steps and the start of the job ID, e.g. 4b-mine.jsonl-100steps-1a2b3c4d; not unique, there is no rename), base_model, model_size, dataset, dataset_sha256, num_examples, num_train_steps, final_loss, params, path, files, size_bytes, created_at. params and files are JSON text on both SQLite and PostgreSQL. job_id is unique and the insert ignores a duplicate, so a replayed completed event (a restart) makes no second row.
  4. A completed event without model (a Modal app that was not redeployed) records nothing and logs a warning. Existing databases get the table the next time the server starts (CREATE TABLE IF NOT EXISTS); there is nothing to migrate.

​
How inference runs

  • model_path is trained/<job_id> and nothing else: the web API refuses anything with another shape (400), and resolve_model_dir checks again inside the container, so a path cannot leave the trained/ directory. A directory without adapter_config.json is a “No trained model at …” error. Without model_path the legacy finetuned_model_<size> adapter is used.
  • model_size must match the weights. It chooses the GPU function (4b: A10G, 12b: A100, 31b: A100 80GB). The Go endpoint passes the size recorded for the model; a direct caller of /infer/query has to pass the right one, or the wrong GPU may run out of memory.
  • The model stays warm. Since gligabao 0.39.0 the first request of a GPU container loads the base model from the HF cache (4-bit) and attaches the adapter (a cold request takes minutes), and the container keeps it in memory and stays up for SIDECAR_INFER_IDLE_TIMEOUT_S (default 60 seconds) after its last request, so the next requests skip the load. A request for another model replaces the loaded one (one model per container). The answer says which it was (cold_start, load_seconds), and the model’s request history in the admin UI shows it for every request. What a cold request does in the container, step by step, and how to read the numbers: Cold and warm starts. The admin UI’s Stop inference button ends the warm containers at once; see Warm inference containers. The Modal functions have a 600 second timeout per request (app.py; the model load counts); the Go client waits up to 15 minutes (context limit, no HTTP timeout).
  • The answer is {"response", "model_size", "model_path", "adapter", "cold_start", "load_seconds", "input_tokens", "output_tokens"} (load_seconds: how long the model load took, the container start not included; 0 when warm). Generation samples (do_sample) with the given temperature (default 0.7) up to max_tokens (default 1024; the Go endpoint allows at most 8192). The prompt is wrapped in the same Gemma turn markers as the training data (prompt_format.py).
  • Errors. The Go endpoint answers 400 for a bad body, 404 for an unknown model, 502 if the sidecar fails (with its message when the generation itself failed, such as missing weights; a generic message otherwise) and 503 without a sidecar. The sidecar reports a failed generation as 200 {"error": "…"}, which the Go client turns into *sidecar.InferError.

​
Adding an inference endpoint

There are three layers, and a new endpoint touches one or more of them. Redeploy the Modal app after a change to the Python side (the training view’s Redeploy), or the running app keeps the old code.
  1. A new Go endpoint over trained models (no Python change; for example a batch call, or a different default):
    • write the handler next to InferOf in internal/api/training/models.go: pick the database (pick), read the model with relational.GetTrainedModel, and take ModelSize and Path from the record, never from the request;
    • if it needs something new from the sidecar, add a method to internal/sys/sidecar/inference.go (a long call uses postVia(ctx, c.stream, …), which has no overall timeout) and to the Inferer interface in models.go (the fake sidecar of the tests implements it);
    • register it in registerTraining (internal/sys/server/server.go);
    • test it with the fake sidecar (internal/api/training/models_test.go).
  2. A new sidecar route (a new kind of request to the Modal app):
    • add ("METHOD", "/path") to ROUTES in sidecar/routes.py: the sidecar forwards the route to its backend untouched, it does not parse the body;
    • add the handler to create_api in sidecar/ai_train/api/api.py: validate the input (bad_model_size, valid_model_path), call an injected function with run_in_threadpool;
    • give create_api that function from serve_api in app.py, built with provider.call(...) (the API module never imports a provider);
    • add the Go client method (step 1) and a test in ApiTest (test_ai_train_core.py) with a fake in place of the function.
  3. A new generation function (another GPU function, e.g. another decoding mode or model type):
    • put the logic in ai_train_core/inference.py (heavy imports inside the function; ai_train_core never imports a provider SDK) and resolve any path with resolve_model_dir;
    • register it in app.py with @provider.function(FunctionSpec(image=INFER_IMAGE, gpu=CONFIG.gpu_for(size), timeout_s=600, volumes=INFER_VOLS, secrets=secrets)): the volumes must include the models and HF cache volumes, and the function must reload the models volume before it reads;
    • route to it from a CPU function like generate, which picks the function by model_size.
A different kind of weights (another model size, or a trainer that saves something other than a LoRA adapter) also needs (see Providers and extending) its own describe result and loader; the row’s path and files already hold any set of files under trained/<job_id>.

​
Details and limits

  • Listing and reading are plain database calls and need no sidecar; only inference does.
  • The row is the record of what was trained, not proof the weights still exist: gligabao does not check the volume.
  • No endpoint renames or deletes a model, merges adapters, or streams the generated text (a model is handed to your own code by a deployment); one request is one prompt and one answer.
  • Training the same settings twice makes two models (and two directories); compare them by final_loss, dataset_sha256 and params.

​
Endpoints

Each model has an Audit button (see Training data audit; its results are in gb_training_audit_runs and gb_training_audit_cases) and an Eval button in the training view of the admin UI (see Evals of trained models); the eval’s results are in the eval tables (gb_experiment_case_runs, gb_training_eval_run_meta and gb_training_case_meta, see Evals of trained models). The columns of gb_trained_models are described in Training jobs. The training view of the admin UI lists the models; clicking one shows its settings and files and a form that runs a prompt through it, with the history of the requests under it. Each request there shows whether it was a cold start (and the model load time) or warm, the time without the load and the token counts. From Go (relational package): RecordTrainedModel, GetTrainedModel, ListTrainedModels; the Go client has Infer(ctx, sidecar.InferRequest).