Skip to main content
POST /v1/gb/remember also remembers images. An image goes through a pipeline of its own, separate from the text pipeline (Remember): a vision LLM reads the text in the image (OCR: its title and text blocks) and classifies the whole image by a theme (plot, graph, …). The image file is kept on the file system, tracked by an image artifact row (metadata and the file’s path, never the bytes), and what the model read is stored as an image extraction, in their own tables. For now nothing is added to the graph: no entities, edges or vectors come out of an image, and query does not see it. Grounding the extractions in the rest of the graph is planned; the extraction tables are the input for it. Code: internal/memory/memory_image/image.go (RememberImage, ExtractImage), internal/memory/memory_image/image_prompts.go, internal/api/remember/remember.go (the handler), internal/memory/memory_image/image_store.go (the files), internal/db/graph/{sqlite,postgres}/db_image.go (SQL).

​
Request

  • image: the image, base64-encoded (standard alphabet): a JPEG, PNG, GIF or WebP, at most 5 MiB (the Anthropic API limit) before encoding. The type is detected from the bytes. A request with image is an image request; the body may be up to about 7 MiB (instead of 1 MiB for text).
  • user_id (required), dataset (optional, default main, created if missing) and reference_time (optional, RFC 3339, default now: when the image was captured or observed) work as for text.
  • session_id (optional) works as for text (Sessions): it is stored on the new image artifact (gb_image_artifacts.session_id) and echoed as session_id in the response. Images add no entities or edges, so there are no session links beyond it. 400 if it is too long or has control characters.
  • models (optional): {"extract_image": "<model>"} picks the vision model that reads the image; without it the default model is used (see Remember). A repeated image is answered from its stored extraction without an LLM call, so the model reported is the one that made it.
  • text and kind must not be given together with image (400): an image is not a text artifact.

​
Response

200 OK:
path is where the file is, relative to the image directory. cached is false here and true for an already processed image. pipeline is always image; it tells this response from the text one. text_blocks is always an array. In eval mode (POST /v1/gb/eval/remember) the response also carries the call’s ATIF trajectory, as for text; the trajectory notes the image ([image: image/png, 20481 bytes]) instead of embedding it.

​
Repeated images

The SHA-256 of the decoded image is compared with the image artifacts the dataset already holds. If one has the same hash, the image is not read again: remember returns the stored result of the earlier call (its newest extraction) with "cached": true, makes no LLM call and stores nothing new (no artifact, extraction, text block or gb_inferences row). In the response, image_artifact_id, session_id, extraction_id, path, valid_at and created_at are those of the stored artifact (so session_id is the session that first stored the image, not necessarily the one of this request; the repeat leaves no trace of its own session) (a reference_time in the repeated request is ignored), and model, provider, input_tokens, output_tokens and cost_usd describe the call that made the extraction, which was spent then, not now. If the image file has gone missing from the image directory, it is written again from the uploaded bytes. Only the same dataset of the same user is searched: datasets are isolated, so the same image in another dataset (or another user’s) is read and stored again. The comparison is on the bytes: a re-encoded or resized copy is a different image. Two identical requests that arrive at the same moment may both miss and both be read. In an eval trajectory a repeat is a single find_existing_image step instead of the LLM and store steps.

​
What the model returns

One LLM call (Complete with the image as an image content block, max 8192 output tokens) with the default model, claude-haiku-5-5 (or the extract_image model of the request’s models), whose price is in the built-in price table. It is asked for only JSON:
Code fences or prose around the JSON are tolerated. Then it is normalized:
  • theme is lowercased and must be one of plot (a data plot with axes), chart (pie, donut, …), graph (nodes and links), diagram, table, document, screenshot, slide, map, photo, illustration or other; anything else becomes other.
  • title is the image’s own title, "" if it has none. If the model gave none but returned a block of kind title, that block’s text is used.
  • text_blocks are the text regions in reading order, each with a kind: title, subtitle, heading, paragraph, caption, label (a drawn element’s text), axis, legend, annotation, table (a row, cells joined by |) or other (the fallback for an unknown kind). Blank blocks are dropped.
  • text is not asked of the model: it is the blocks joined by newlines.
The call is recorded in gb_inferences with type remember_image (stage extract_image, with the reasoning level it ran with: low unless the request sets reasoning.extract_image) (the text pipeline’s calls are remember), even if a later step fails.

​
Where the files are

Image bytes are never stored in the database. They are files in the image directory, <db path without extension>.images next to the SQLite file (e.g. build/gligabao.images, or /data/gligabao.images in the container). Override it with serve --image-dir <path>, or ServeOptions.ImageDir when embedding; the directory is created when the first image is stored. In Go, memory_image.ImageDir is the default and memory_image.WithImageDir(dir) overrides it for one call. A file is <dataset id>/<sha256>.<ext> (jpg, png, gif or webp) inside it, and gb_image_artifacts.path holds that relative path, so the directory can be moved. The name is the SHA-256 of the content, which is also how a repeated image is recognised (see Repeated images). The file is written first (atomically, via a temporary file and a rename) and the rows after; if storing the rows fails, a file that this call created is removed again. Only a crash between the two steps can leave a file that no row tracks. With PostgreSQL the files still go to the local directory, so the directory must be on storage the server keeps (a volume, in the container). memory_image.LoadImage(ctx, db, userID, dataset, imageArtifactID) reads a stored image back.

​
Schema

Created on startup in the same database as the other tables (internal/db/graph/{sqlite,postgres}/db_schema.go). Both rows and all blocks of a call are written in one transaction, after the LLM call: if it fails nothing is stored (and a dataset that did not exist is not created). Each new image adds one image artifact (a repeat adds nothing, see above); an artifact can have several extractions (for example from different models) but remember makes one. Image artifacts are not in gb_artifacts and do not show up in the graph export. Go access, in package graph: InsertImage, ListImageArtifacts, GetImageArtifact, ListImageExtractions.

​
Limitations

  • No endpoint reads, lists or deletes image artifacts or extractions yet; use the database.
  • Not grounded: entities, edges and embeddings are not derived from the extraction, and query ignores images.
  • The files are on the local file system only (no object storage), and nothing deletes them: deleting a dataset (a soft delete) keeps its image rows and files, and there is no endpoint to remove an image.
  • The theme is the model’s single best guess; there is no confidence value and no bounding boxes for the text blocks.
  • Huge images are not resized, and a dense image can exceed the output limit (502).