Hooks

​

System events

import "github.com/deepblue-phoenix/gligabao"

err := gligabao.Serve(ctx, cfg, gligabao.ServeOptions{
    DBPath: "data/gligabao.db",
    Hooks: []gligabao.Hooks{{
        SystemEvent: func(ctx context.Context, eventType string, ev gligabao.SystemEvent) {
            log.Printf("%s: %v", eventType, ev)
        },
    }},
})

​

Types

type Hooks = monitoring.Hooks
type SystemEvent = monitoring.Event // map[string]any

// in package monitoring:
type SystemEventHook func(ctx context.Context, eventType string, event Event)
type Hooks struct {
    SystemEvent SystemEventHook // nil: skipped
}

​

Behaviour

• Installed for the lifetime of Serve. Serve installs ServeOptions.Hooks when it starts and removes them when it returns. There is no way to add or remove a hook while the server runs.
• Several hook sets. ServeOptions.Hooks is a list; every set is called for each event, in the order of the list. A nil field is skipped. With no hook installed, events are dropped.
• Synchronous. The callback runs on the goroutine that handles the request, before the HTTP response is written, so a slow callback slows the response. Keep it fast; hand slow work (network calls) to a goroutine or a buffered queue. (gligabao_int does this for its BetterStack sink.)
• Concurrent. Requests are handled concurrently, so the callback must be safe for concurrent use.
• Read-only event. The SystemEvent map must not be modified, and should not be retained after returning if you may change it: copy it first.
• Panics are contained. A callback that panics does not affect the request or stop the remaining callbacks; the panic is silently discarded, so recover and log yourself if you want to know.
• ctx is the context of the HTTP request that caused the event (cancelled if the client leaves).
• Events are emitted by the regular endpoints and by their eval variants (POST /v1/gb/eval/remember, POST /v1/gb/eval/query), which use the same handlers.

​

Event types

​

remember.completed

​

remember.failed

​

remember_image.completed

​

remember_image.failed

​

query.completed

​

query.failed

​

timeseries.completed

​

timeseries.failed

​

Example: metrics and alerts

SystemEvent: func(_ context.Context, typ string, ev gligabao.SystemEvent) {
    ms, _ := ev["duration_ms"].(int64)
    requestDuration.WithLabelValues(typ).Observe(float64(ms) / 1000)
    if strings.HasSuffix(typ, ".failed") {
        alert(typ, ev["user_id"], ev["error"])
    }
},

​

Stability

​

Timeseries providers

gligabao.ServeOptions{
    TimeseriesProviders: []gligabao.TimeseriesProvider{{
        Name:        "prices",
        Description: "Daily stock prices. Use for publicly traded companies.",
        QueryFormat: "the ticker symbol, e.g. AAPL",
        Fetch: func(ctx context.Context, req gligabao.TimeseriesFetchRequest) ([]gligabao.TimeseriesData, error) {
            /* ... */
        },
    }},
}

​

Trained model deployers

gligabao.ServeOptions{
    TrainedModelDeployers: []gligabao.TrainedModelDeployer{{
        Name: "bucket",
        Description: "Uploads the model's files to the releases bucket.",
        Deploy: func(ctx context.Context, req gligabao.DeployTrainedModelRequest) (gligabao.DeployTrainedModelResult, error) {
            // req.Dir has req.Model.Files: adapter_config.json, adapter_model.safetensors, ...
            return gligabao.DeployTrainedModelResult{Ref: "models/my-adapter", Details: map[string]any{"job_id": "..."}}, nil
        },
    }},
}

​

Behaviour

• gligabao fetches the files; the deployer does not. The weights live on the sidecar’s storage (Modal: the models volume). Before calling the deployer gligabao downloads every file of Model.Files from the sidecar (GET /models/file, see Sidecar training API) into Dir, and checks each against its recorded size and sha256. The deployer gets files that are complete and verified, and never talks to the sidecar.
• Dir is an absolute path, <trained model deploy dir>/<job id> (serve --trained-model-deploy-dir, ServeOptions.TrainedModelDeployDir, default trained_models next to the database file). The files are at the paths Model.Files lists, relative to it (adapter_config.json, adapter_model.safetensors, tokenizer.model, …). gligabao never deletes it: it is kept for later deployments of the same model, which find the files in place (verified again) and download only what is missing or damaged. Clean it up yourself. The deployer must treat it as read only; copy or archive what it needs.
• Synchronous. The request that asked for the deployment waits for the fetch and then for Deploy; its HTTP response is the stored deployment. Fetching a large model and a slow upload can take minutes: callers must not use a short timeout. ctx is the request’s context (cancelled if the client leaves), so Deploy should honour it.
• Result. On success Ref identifies what the deployer made, in its own terms (e.g. the name of the uploaded model), and Details is anything else worth keeping; it must be JSON-encodable. Both are stored and returned with the deployment.
• Failure. An error returned by Deploy (or a panic, which is caught) marks the deployment failed with the error text and answers 502. So does a failure to fetch the files. A failed deployment is not retried by gligabao.
• Concurrent. Requests are handled concurrently, so Deploy must be safe for concurrent use, also for the same model. DeployID is unique to each call: use it if the deployer needs a unique name.
• Recorded. Every deployment that reaches the fetch is stored in gb_trained_model_deployments (status running, then completed or failed; the deployer, its options, the directory, ref, details, the error, the duration) and listed by GET /v1/gb/training/models/{id}/deployments. A row still running long after it started was interrupted (the server restarted).
• Validated at startup. Serve returns an error for a deployer with no Deploy, a Name that is not 1-64 letters, digits, _, - or . (starting with a letter or digit), or a duplicate name. At most 20.
• System events. Each deployment emits training.model_deploy.completed or training.model_deploy.failed to the SystemEvent hook, with model_id, deploy_id, deployer, duration_ms (int64) and, on failure, error.