Timeseries design proposal

Status: v1, v2 and the series_observation part of v3 are implemented; embedding series and query integration (the rest of v3) are not. v2 (providers and the fetch stage) differs from the sketch below: see Changes made while building v2; the observations differ too: see Changes made while building v3. v1: What exists is described in Timeseries: supplied series, anchors (artifact, entity, edge) resolved after Store, stats, the POST /v1/gb/timeseries endpoint and two read endpoints. This file keeps the reasoning behind it and the plan for what is left: the provider fetch stage (v2), and derived series_observation artifacts (v3, done), then embedding and query (v3, still to do). Where the text below differs from Timeseries, that page is right (see Changes made while building v1).

​

Summary

1. Supplied series. The caller sends points in the request. Anchoring is resolved after extraction, so it works although the entities do not exist yet when the request is made.
2. Fetched series. An opt-in enrichment stage after Store resolves entities to series through a provider (for example entity to ticker to price series). It follows the web_search pattern.
3. A standalone attach endpoint. It covers “I only know the entities after remember returns”, refreshes live series, and takes large uploads. remember uses the same code path internally.

​

The timing problem

​

Data model

gb_series          id, dataset_id, key, name, unit, frequency,
                   source ('supplied' | 'fetched'), provider,
                   definition (JSON: symbol, field, params; enough to refetch),
                   first_ts, last_ts, point_count, created_at
gb_series_points   series_id, ts, field, value          -- PK (series_id, ts, field)
gb_series_anchors  series_id, anchor_type ('artifact' | 'entity' | 'edge'), anchor_id,
                   role, window_start, window_end, created_at
gb_series_stats    series_id, window, min, max, mean, last, pct_change, trend,
                   change_points (JSON)

• Many-to-many anchors, like gb_entity_artifacts. One price series can anchor to an entity, to several news artifacts, and to the edges that describe events.
• The window lives on the anchor, not on the series. What matters for memory is “the price in the 30 days around this news item”. The series is shared; each anchor says which slice matters.
• Series identity is (dataset, provider, key, field, frequency). Re-sending or re-fetching upserts points (as the image SHA-256 deduplicates images).
• field supports OHLCV and multi-metric series without a table per shape.
• Bi-temporal points. ts is event time. An optional recorded_at can be added later for restated data (macro revisions). It is not in v1, but nothing should be designed to exclude it.
• gb_series_stats holds deterministic summaries computed in Go: min, max, last, percent change, trend, change points. These are what extraction, embedding and query see. The LLM never sees raw points.
• Embed the description plus the stats, never the points, for example “AAPL daily close, Jan 2025, -6.2%, largest drop on 2025-01-31”.

​

Request

{
  "user_id": "u1",
  "text": "Apple fell sharply after the earnings miss.",
  "kind": "episode",
  "reference_time": "2025-01-31T21:00:00Z",

  "timeseries": {
    "series": [{
      "key": "AAPL.close", "name": "Apple close", "unit": "USD", "frequency": "1d",
      "points": [["2025-01-29T00:00:00Z", 235.1], ["2025-01-30T00:00:00Z", 237.6]],
      "anchors": [{"entity": {"name": "Apple", "type": "organization"}, "on_unmatched": "create"}]
    }],
    "fetch": {"providers": ["prices"], "window": "30d"}
  }
}

• timeseries.series[] (supplied): points are [ts, value] pairs to keep the JSON small. A series without anchors is anchored to the artifact. The 1 MiB body limit applies, so there is a cap on points per series; large series go through the attach endpoint.
• timeseries.fetch (fetched): opt-in, like web_search. It costs money and time and can fail, so nothing is fetched by default.
• models.resolve_series (optional): the model for symbol resolution, if an LLM is used for it.
• Errors: 400 for malformed points, timestamps or anchors; 503 if fetch is asked for and no provider is configured.

​

Pipeline placement

1 Extract -> 2 Resolve edges -> 3 Store (+ anchor supplied series)
          -> 4 Embed -> 5 web_search -> 6 timeseries fetch (new)

• Supplied series are stored inside step 3’s transaction. Entity and edge selectors can only be resolved once the rows exist, and a bad series should fail the request before commit. Nothing is fetched, because the points are in the request.
• Fetched series run after the commit, like web_search. A provider failure never fails remember; it is reported in the response and the eval trajectory.

​

The fetch stage

1. Pick key entities, reusing the web-search heuristic: skip concept, rank by number of edges of the call, cap at about 3.
2. Resolve: ask each requested provider whether it supports the entity and which series it maps to.
3. Fetch the window around reference_time (an event-study shape: before, at and after the event), and optionally the latest window.
4. Store: upsert the series and points, add anchors, compute stats.
5. Embed the series description and stats.

​

Providers

type Provider interface {
    Name() string
    Resolve(ctx context.Context, e Entity) ([]SeriesRef, error) // nil: not applicable
    Fetch(ctx context.Context, ref SeriesRef, from, to time.Time) ([]Point, error)
}

• Entity to ticker is where this goes wrong. A guessed ticker for “Apple” can be the wrong company. Prefer precision over recall: require an unambiguous match and skip otherwise.
• Cache the resolved identifier on the entity (an identifier attribute, or a has_ticker edge), so it is resolved once.
• Providers by entity type. Once ontologies are used by remember, a class (Company) can imply a provider.

​

The attach endpoint

POST /v1/gb/timeseries
{"user_id": "u1", "dataset": "main", "series": {...},
 "anchors": [{"entity_id": 12}, {"artifact_id": 1, "window": ["2025-01-01", "2025-02-01"]}]}

• Anchors by ID or selector to things that already exist, such as the entity names returned by an earlier remember.
• Has a fetch mode: {"entity_id": 12, "provider": "prices"}.
• Refreshes live series. remember should not own ongoing updates.
• Accepts bigger bodies (or a file upload) than remember.
• remember is defined as “extract, then attach”, so there is one code path.

​

Making series part of the graph

• A new artifact kind, series_observation, with parent_id pointing at the anchoring artifact and a definition or query field.
• Its text is generated deterministically from the stats (“AAPL closed down 6.2% on 2025-01-31, its largest one-day drop in 90 days”).
• That text goes through the normal extract, reconcile and store path, so it yields bi-temporal edges, deduplication and embeddings with no new logic.

​

Changes made while building v1

• No field column on points. field is a column of gb_series and part of the series identity (provider, key, field); an OHLCV series is several series with the same key. Points are (series_id, ts, value). frequency is only a label and is not part of the identity.
• Stats are a JSON column of gb_series (stats), over all the points, instead of a gb_series_stats table with a window. A window’s stats are computed on read (GET /v1/gb/timeseries/{id}?from=&to= returns window_stats). Change points are replaced by the single largest_move.
• Anchors have one real foreign-key column per target type (artifact_id, entity_id, edge_id, exactly one set) instead of anchor_type and anchor_id, so deleting a target cascades to its anchors.
• on_unmatched defaults depend on the anchor: create for an entity selector, ignore for an edge selector, error for an ID. create is not allowed for an edge.
• The attach request takes series as a list, each with its own anchors, like remember, rather than one series with top-level anchors. Its dataset must exist (404); a series without anchors is stored unanchored.
• Not done in v1: embedding the series description and stats (the entity that an anchor creates is not embedded either), and the models.resolve_series stage (there is no LLM step). timeseries.fetch is a 400.
• Chunked text: the series are stored after the chunks, anchored to the parent artifact, in a transaction of their own, so they are not atomic with the chunks.
• Lookahead leakage (risk 1): series are stored as supplied and never clipped at reference_time; clipping is left to the caller until a fetch stage exists.

​

Changes made while building v2

• Providers are registered by the embedding program, with ServeOptions.TimeseriesProviders, not configured: a provider is {Name, Description, QueryFormat, Fetch}. The sketched Resolve method is gone: an LLM decides which provider is used for which entity (the resolve_series stage), from the natural-language Description of each provider, and writes the provider-specific query (a ticker) from its QueryFormat. Fetch receives the entity, that query and the period.
• No entity-to-identifier cache yet: the decision is repeated for every request.
• Entities offered: up to 20, by number of facts, with no type excluded (a provider may be for concepts). At most 10 fetches per request.
• Lookahead (risk 1) is decided: a fetch reaches before (default 30 days) before the reference time and after (default none) after it, and what a provider returns is clipped to that period.
• Anchors: the fetched series is anchored to the entity and to the artifact the entity came from, both with the period as the window.
• The standalone endpoint has no fetch mode yet.

​

Changes made while building v3

• Opt-in with timeseries.observations, at most 10 per request. The sketch did not say; each observation is an extraction call, as a web search is a search, so nothing is derived by default.
• It describes what the request wrote (the period of the supplied points, or the fetched period), not the whole stored series, and uses the stats of those points computed on read in Go (Summarize), not the stored gb_series.stats. “Its largest one-day drop in 90 days” is therefore not said: the text only knows the window it was given (largest move between consecutive points, high, low, average, change, trend).
• The text names a subject: the entity of the series’ first entity selector (supplied), or the entity it was fetched for. Without it the extraction has nothing to attach the facts to.
• A column for the series: gb_artifacts.series_id (→ gb_series, ON DELETE SET NULL), so the chain fact → observation → series is a real link and not a convention; query says which series and window. The sketch’s “definition or query field” is query. The observation is a child of the artifact that anchors the series: the call’s artifact (the parent, for a chunked text) for a supplied series, the artifact the entity came from for a fetched one. valid_at is the last point of the window.
• Last stage, after the web search and the fetch (it needs the fetched series), and like them it never fails remember.
• Not done: a kind-specific treatment in query (it sees an observation’s facts like any other fact), an observation by the standalone endpoint, deduplication of observations.

​

Rollout

• internal/memory/memory_timeseries/ (parse, anchor, stats, fetch)
• internal/db/graph/{sqlite,postgres}/db_timeseries.go
• internal/api/timeseries/
• docs/timeseries.md (now Timeseries)

​

Risks and open questions

1. Lookahead leakage. If a series extends past reference_time, a memory of an event contains the future. Either clip fetched windows at reference_time, or label the post-event part explicitly. Decision needed.
2. Scale. Points in SQL tables are fine to millions of rows. A dedicated store (Timescale, Parquet files like the image directory) is an alternative. Proposed: start with SQL.
3. Licensing and rate limits. Market data providers restrict storing and redistributing data.
4. Scope of v1. Supplied-only first, or fetch immediately? Is there a specific finance provider or key? Decision needed.
5. Deletion. Deleting an anchor artifact or entity should remove its anchor rows. A series that other anchors still use should stay; one with no anchors can be removed.
6. Time handling. Timezones, irregular sampling and units need explicit rules (UTC RFC 3339 only, as everywhere else; frequency is a label, not a constraint).