Make usage explainable
AI Token Usage Cache Index
Build an accounting trail from each request to the usage it actually reports.
The idea, in context
An AI token usage cache index is a catalog of measurement records, not a live inventory of every token in a model. It connects a request to model identity, reported input and output usage, cache-related fields, and the definitions used to interpret them. Keep the raw provider response as well as any normalized values. That makes later comparisons possible when a field, model, or pricing rule changes. Distinguish the observation itself from an estimate of what it might cost.
Retain definitions with the data
Document whether cached input is a subset of an input total or a separate field under the selected provider’s accounting. Preserve absent fields as unknown rather than converting them automatically to zero. A record should explain both its values and its limits.
Account for the whole workflow
A user task can involve retries, retrieval calls, tool exchanges, or more than one model request. Keep a task identifier and individual request identifiers. Deduplicate observations before summing them, then describe which parts of the workflow your totals cover.
Compare a controlled workload
Choose representative tasks and hold the evaluation conditions steady. Look at quality, latency, and usage together. If you estimate savings, label the rates and workload assumptions and separate the hypothetical comparison from an actual invoice.
Questions about ai token usage cache index
Is hit rate enough to estimate savings?
No. Hits can reuse very different quantities of input, and a workflow can have additional costs. Use the reported quantities and the actual applicable rates.
What should happen when a usage field is missing?
Preserve that uncertainty. Record the field as unavailable and avoid treating missing data as evidence of no usage or no reuse.
Start with one good question.
Follow a topic, explore the tradeoffs, and go deeper in the Lab.