Python sidecar

sidecar/, sidecar/routes.py, sidecar/backends/, sidecar/sidecar_config.py, sidecar/ai_train/, Dockerfile.sidecar, docker-compose.yml, run_build_sidecar.sh, internal/sys/sidecar/sidecar.go, internal/sys/sidecar/training_jobs.go, internal/sys/sidecar/training_events.go The sidecar is a small Python service that runs next to the Go server. The Go server only ever talks to the sidecar, and the sidecar proxies to the compute backend (Modal today). The Go code never calls Modal or any other provider, never holds provider credentials and never learns what runs behind the sidecar’s REST API.

Go server ── HTTP ──► sidecar (sidecar/, FastAPI) ── HTTP ──► backend: web API of the Modal app (sidecar/ai_train)
internal/sys/sidecar        routes.py → Backend.forward              runs the jobs on a cloud GPU

It exists for work that is easier to do in Python (libraries that have no good Go equivalent), and for keeping everything cloud-specific out of the Go server. The sidecar answers GET /health itself and forwards every other route, unchanged, to its backend. The backend that exists today, modal, runs Gemma-4 fine-tuning and inference with its own web API; see Remote training. The Go server starts training jobs through the sidecar (the endpoint POST /v1/gb/training/jobs, or the training view of the admin UI), records each one and follows its progress over SSE; this is documented in the Training section.

The service

sidecar/main_sidecar.py is a FastAPI app run by uvicorn. Dependencies are in sidecar/requirements.txt (including httpx and modal).

Endpoint	Response
`GET /health`	`200 {"status": "ok"}`, answered by the sidecar itself. It says the sidecar is up, not that the backend is reachable.
`POST /training/job_start`, `GET /training/job_progress`, `GET /training/job_events` (SSE), `POST /infer/query`, `POST /evals/add_to_dataset`, `GET /evals/get_dataset_count`	Answered by the backend (below).

Backends

The routes in sidecar/routes.py are the only API the Go server knows (internal/sys/sidecar/). Each one is forwarded as it is to the configured Backend (sidecar/backends/): method, query string, body and content type go out; the status, content type and body come back as the backend sent them, and an SSE stream such as job_events is passed through as it arrives, with no read timeout. A 4xx or 5xx from the backend reaches the Go server unchanged. If the backend cannot be reached at all, the sidecar answers 502 {"error": ...}. The only backend today is modal (backends/modal_backend.py): it forwards to the web API (serve_api) of the Modal app in sidecar/ai_train/app.py. Deploy that app from the admin UI (the sidecar runs the deploy itself and streams its log; see Modal app lifecycle) or with cd sidecar/ai_train && modal deploy app.py; the sidecar finds its web URL through the Modal SDK, once, on the first request (the app must be deployed, and the app name must be SIDECAR_APP_NAME). With modal serve (dev) there is no deployed app, so set SIDECAR_MODAL_WEB_URL to the URL it prints. A URL that cannot be found is a 502 that says so. To add a backend, implement Backend.forward in backends/ and register it in backends/__init__.py::get_backend; an HTTP service that serves the same API can subclass HttpBackend and only provide its URL. The Go server needs no change. Besides forwarding, the sidecar answers a few routes itself, for what an operator does to a backend (sidecar/backends/actions.py). The modal backend uses them to manage the Modal app’s lifecycle without the modal CLI: deploy (the Modal SDK’s app.deploy() in a subprocess), stop (modal app stop), cancel of a running one, a live log of each run, a read-only list of the active Modal apps of this system (the configured app and <it>-<suffix> ones), each with its own deploy, stop and log, and a read-only list of the app’s volumes. The credentials are the sidecar’s MODAL_TOKEN_ID / MODAL_TOKEN_SECRET; the deployment itself lives in Modal, so it survives sidecar restarts (only the last run’s log is kept in memory). The admin UI’s training view drives it. How it works, the stages, the log stream, what Stop does to running jobs and troubleshooting are on Modal app lifecycle.

Endpoint	Response
`GET /backends`	The backends with their live status (`name`, `configured`, `healthy`; for `modal` also `deployed`, `app`, `error`, and each action’s state).
`POST /backends/{name}/actions/{action}`	Starts the action (`deploy`, `stop`): `202` with its state, `409` if it (or the action it excludes) is running, `404` for an unknown backend or action.
`POST /backends/{name}/actions/{action}/cancel`	Stops a running action: SIGINT, then a kill after 10 seconds. `202`, or `409` if it is not running. The run ends as `cancelled`.
`GET /backends/{name}/actions/{action}[?from=<i>]`	The state (`idle`, `running`, `succeeded`, `failed`, `cancelled`; `stage`, `stages`, `started_at`, `finished_at`, `exit_code`, `error`) and the kept log (last 5000 lines, stdout and stderr merged).
`GET /backends/{name}/actions/{action}/events[?from=<i>]`	SSE: `log` and `state` events; ends when the run is over.
`GET /backends/{name}/apps`	`{"apps": [...]}`: the active apps of this system with the state of their actions. `404` if the backend has none, `502` if they cannot be listed.
`…/backends/{name}/apps/{app}/actions/{action}[/events\|/cancel]`	The action routes above for one app (`404` for an app that is not ours).
`GET /backends/{name}/volumes`	`{"volumes": [...]}`: the backend’s storage volumes, read only. `404` if it has none, `502` if they cannot be listed.

The Go server passes them on, unchanged, as GET /v1/gb/admin/api/sidecar/backends (the list) and POST|GET /v1/gb/admin/api/sidecar/backends/{name}/actions/{action}[/events|/cancel] GET .../backends/{name}/volumes, GET .../backends/{name}/apps and .../backends/{name}/apps/{app}/actions/{action}[/events|/cancel]. Because the sidecar has no authentication, keep it on localhost or a private network: anyone who reaches these routes can deploy or stop the app. A new backend gets actions by overriding Backend.actions() (and apps(), app_actions(), volumes()); the admin UI draws the buttons it reports.

Configuration

Configuration is by environment variable. At startup main_sidecar.py reads them once into a SidecarConfig (a frozen pydantic model, sidecar/sidecar_config.py) and passes it to create_app(config); the app keeps it on app.state.config (handlers can use the get_config helper). New settings are added as fields of that struct. An invalid value stops the sidecar at startup with a validation error.

Variable	Default	Meaning
`SIDECAR_HOST`	`127.0.0.1`	Interface to listen on. Local only by default, as the sidecar has no authentication.
`SIDECAR_PORT`	`8787`	Port to listen on.
`SIDECAR_BACKEND`	`modal`	Which backend answers the API (only `modal` exists).
`SIDECAR_APP_NAME`	`gligabao-train`	Modal backend: name of the deployed Modal app. The training code reads the same variable, so the two agree.
`SIDECAR_MODAL_WEB_URL`	(unset)	Modal backend: use this URL instead of looking the app up.
`MODAL_TOKEN_ID`	(required for `modal`)	Modal API token id.
`MODAL_TOKEN_SECRET`	(required for `modal`)	Modal API token secret.

With the modal backend the credentials are checked when the config is loaded: the sidecar refuses to start if either is missing or empty (they are held as SecretStr, so they do not show up in logs). Create a token with modal token new; a ~/.modal.toml is not read by this check. They live only in the sidecar’s environment: the Go server never sees them. Run it without a container:

cd sidecar
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
export MODAL_TOKEN_ID=ak-... MODAL_TOKEN_SECRET=as-...
python main_sidecar.py              # listens on 127.0.0.1:8787
curl http://127.0.0.1:8787/health

Remote training

sidecar/ai_train/ fine-tunes Gemma-4 (4B, 12B and 31B, LoRA) on a cloud provider and serves the result, with its own web API. It is the Modal side of the modal backend above: it runs on Modal, not inside the sidecar process, and is not imported by main_sidecar.py; Dockerfile.sidecar copies it only so that the sidecar can deploy it to Modal. Nothing but the sidecar calls its web API. It has its own section of this site, Training:

Training jobs: how the Go server starts a job and records it in gb_training_job_runs, and the TrainingJob struct.
Training datasets, Model sizes and the Sidecar training API.
Running on Modal: the code layout, the modal commands and the SIDECAR_* settings of the training code.
Modal app lifecycle: deploy, live log, cancel, stop and volumes from the admin UI.
Providers and extending: the provider abstraction, adding a provider, a model size or a function.

The container

Dockerfile.sidecar (in the repo root; the plain Dockerfile is the Go server’s) packages the sidecar on python:3.13-slim: it installs the requirements, copies main_sidecar.py, sidecar_config.py, routes.py, backends/ and ai_train/ (the code the deploy action uploads to Modal), runs as the non-root user sidecar and starts python main_sidecar.py. The image sets SIDECAR_HOST=0.0.0.0 (a container must listen on all interfaces for a published port to reach it) and exposes 8787.

./run_build_sidecar.sh                         # builds gligabao-sidecar:latest
./run_build_sidecar.sh myrepo/sidecar:dev      # or a name of your choice
docker run --rm -p 127.0.0.1:8787:8787 -e MODAL_TOKEN_ID -e MODAL_TOKEN_SECRET gligabao-sidecar:latest

Without the two Modal variables the container exits at startup. Publish the port on 127.0.0.1 (as above) unless the sidecar must be reachable from other machines: it has no authentication.

With docker compose

docker-compose.yml (in the repo root) starts the server and the sidecar on the same network (gligabao):

export ANTHROPIC_API_KEY=...
docker compose up --build

On that network the server reaches the sidecar at http://sidecar:8787 (the service name; 127.0.0.1 would be the server’s own container). The server starts only once the sidecar’s /health check passes. Both ports are published on 127.0.0.1 only; the server’s data lives in the gligabao-data volume. The sidecar needs MODAL_TOKEN_ID and MODAL_TOKEN_SECRET in its environment; docker-compose.yml does not pass them yet, so add them under the sidecar service’s environment: (the server needs none).

The Go client

internal/sys/sidecar is the only package in the Go code that knows the sidecar’s HTTP API, and the sidecar is the only thing the Go server calls for this work: there is no Modal client, token or URL anywhere in Go. In it sidecar.go has the client and Health, training_jobs.go and training_events.go the training calls.

c := sidecar.New("")                       // "" is sidecar.DefaultBaseURL, http://127.0.0.1:8787
resp, err := c.Health(ctx)                 // resp.Status == "ok"

sidecar.New(baseURL) returns a *Client with a 30 second timeout per request. A trailing / on the URL is ignored.
Client.Health(ctx) calls GET /health. It returns an error if the sidecar cannot be reached, answers with a status other than 200, sends a body that is not the expected JSON, or reports a status other than "ok".
Client.StartTrainingJob(ctx, req) calls POST /training/job_start and returns the TrainingJob the sidecar reports; Client.StreamTrainingJobEvents(ctx, jobID, from, fn) follows GET /training/job_events (SSE) and calls fn with each event; StartTrainingJobRequest.DatasetContent carries the dataset file. There are no dataset methods: the sidecar has no dataset endpoints. See Training jobs and Training datasets.
Errors name the method and path (sidecar GET /health: ...) and, for a bad status, include the start of the response body. A bad status is a *sidecar.StatusError (Code, Body, and Message(), the sidecar’s error text), so callers can tell a request the sidecar rejected (400) from a sidecar that is down. The event stream has no overall timeout; its context ends it.

Startup check and status

On startup the Go server calls the sidecar’s GET /health once (5 second timeout). If the sidecar does not answer correctly the server still starts: it logs a warning, sets the server.SidecarActive flag to false and runs without a sidecar client, so everything that needs the sidecar answers 503 (“sidecar is not active”) or does nothing. The check runs once, so restart the server after starting the sidecar. Because /health is answered by the sidecar itself, a sidecar whose backend (Modal) is down still counts as active; its calls then fail with 502 when they are made. GET /v1/gb/admin/api/sidecar/status answers {"sidecar_up": bool, "sidecar_active": bool}: sidecar_up is a live health check made by the server, sidecar_active is the decision taken on startup. The admin UI shows sidecar_up as a badge in the top bar (sidecar: up / sidecar: down, refreshed every 15 seconds; sidecar: up (restart server) when it is up but was down at startup), and its training view uses sidecar_active to show a “Sidecar not active” notice and disable Run. The tests (internal/sys/sidecar/sidecar_test.go) run the client against an httptest server, so they need neither Python nor Docker.

Adding an endpoint

An endpoint has three parts: the Go client, the sidecar route and the backend that answers it.

Add the path to ROUTES in sidecar/routes.py. It is then forwarded to the backend as it is; the sidecar does no work of its own for it.
Implement it in the backend: for modal, a route of the web API in sidecar/ai_train/api/ (see the Sidecar training API). Another backend must serve the same path.
Add a method to Client in internal/sys/sidecar (a new file per topic, like training_jobs.go), with typed request and response structs. Keep all calls to the sidecar in that package, and never call a backend from Go.
If the Go server should start it, add an admin handler in internal/api/admin that takes the client through a small interface, as StartTrainingJobOf does.
A route that is answered by the sidecar itself, like /health, is added in sidecar/main_sidecar.py instead (and any new package to sidecar/requirements.txt). If the Dockerfile must copy new files, update Dockerfile.sidecar.
Document the endpoint in the tables above.

Embedding in Go

Operations

Appendix

The service

Backends

Configuration

Remote training

The container

With docker compose

The Go client

Startup check and status

Adding an endpoint

​The service

​Backends

​Backend actions and Modal lifecycle

​Configuration

​Remote training

​The container

​With docker compose

​The Go client

​Startup check and status

​Adding an endpoint

The service

Backends

Backend actions and Modal lifecycle

Configuration

Remote training

The container

With docker compose

The Go client

Startup check and status

Adding an endpoint