sidecar/ai_train/ai_train_core/model_config.py, model_train.py
Three sizes can be trained, chosen per job with model_size (4b, 12b or 31b).
Comparison
All the numbers are in
ai_train_core/model_config.py (pure data).
Training backend
google/gemma-4-E4B-it (4B) uses the gemma4 architecture and google/gemma-4-12B-it (12B) the newer gemma4_unified one. All train through Unsloth (2026.10.x, transformers<=5.17) in one shared image (requirements_train.txt) and serve through plain transformers + peft + bitsandbytes in a separate, lighter inference-only image (requirements_infer.txt: no Unsloth, trl, datasets or torchao, and a slim base image without the CUDA toolkit). The base model is loaded 4-bit and the trained adapter attached on top. See Images for all the images. Keep the torch and transformers ranges of the two requirements files in step. Every job saves its LoRA adapter in a directory of its own, /models/trained/<job_id>, so no job (of any size) overwrites an earlier model; each completed job is a trained model. Only the adapter is saved: the base model stays in the Hugging Face cache volume and is shared by all jobs, and inference needs model_size to match the size the adapter was trained for.
Images
app.py declares four images (ImageSpec); every function names the one it runs in. Code is mounted last, so a code change never rebuilds the dependency layers.
The inference image is kept as light as possible: it has only
torch, transformers, peft, accelerate, bitsandbytes and modal. No Unsloth (and none of what it pulls in), no trl, datasets, torchao or torchvision, and no CUDA toolkit: the torch wheels bring the CUDA libraries they need and Modal provides the driver (ImageSpec(gpu=True, cuda_toolkit=False)). That makes the build and the cold start of an inference container smaller than with the training image. The adapter a training job saves is a standard peft LoRA adapter, so inference loads the base model 4-bit (nf4) with bitsandbytes and attaches the adapter with peft (model_load_infer.py). Keep the torch and transformers ranges of requirements_train.txt and requirements_infer.txt in step. Redeploy the Modal app to build the new image.
LoRA settings
All use rank 16, alpha 16 and dropout 0 on the q/k/v/o attention projections, with gradient checkpointing and 4-bit quantization of the base model.GPU selection
The default comes fromdefault_gpu in model_config.py: 4b is a10g, 12b is a100 and 31b is a100-80gb. SIDECAR_GPU overrides it for the train and generate functions of every size. The job’s timeout is SIDECAR_TRAIN_TIMEOUT_S (default 7200 seconds). See Running on Modal.
Cost estimates
These are estimates at Modal’s prices when they were written, and include the model load. There is no table for 31b yet: it runs on an A100 80 GB at roughly $5 per hour and is slower than 12b, so plan for a multiple of the 12b figures.
Which one to use
- 4b: lower cost, a smaller dataset (under about 1000 examples), fast iteration, a simple task (tool calling, classification, basic Q&A), quick tests before a larger run.
- 31b: when 12b is not good enough and quality matters more than cost or speed; needs an 80 GB GPU (A100 80 GB).
- 12b: higher quality, a larger dataset (over about 1000 examples), complex reasoning or longer context, production, and a 3 to 4 times higher cost is acceptable.
Troubleshooting
- Out of memory on 4b (A10G): it should not happen with batch size 2. If it does, reduce the batch size to 1 and raise gradient accumulation to 8.
- Out of memory on 12b (A100): it should fit with batch size 1. Make sure 4-bit quantization and gradient checkpointing are on; consider the A100 80 GB.
- Too slow, 4b: an A100 is about twice as fast, at a higher cost.
- Out of memory on 31b (A100 80 GB): keep batch size 1, 4-bit quantization and gradient checkpointing on, and reduce
max_seq_lengthfrom 2048 to 1024. - Too slow, 12b: use the A100 80 GB, or reduce
max_seq_lengthfrom 2048 to 1024.