Grafferene: measure what a response cache would save before it serves an answer

2026 – present · Independent work, unaffiliated with any employer.

What a cache saves depends on how much the traffic repeats. So it measures that first, without serving a single cached answer.

An LLM response cache that starts by watching. A week of real traffic yields a report of would-be hits and the stored replies it would have returned, and the owner decides from it whether, and how strictly, serving begins.

Grafferene drawn as a sphere of graphene cells between an application and its model endpoint: an exact repeat lands on its stored answer, a similar question asked in the same context lands beside its closest neighbors, is checked and gets the stored answer, and everything else passes through to the model. Illustration from grafferene.ca.

The demo as recorded: 2 minutes 3 seconds of the deployed demo, in real time, captions on screen and no sound. 962 recorded WildChat turns from 176 users in three tenants are replayed through Grafferene, each request answered from the cache or sent on to the model with its reason on the record. A question typed into the page is decided in the browser by an 8-bit copy of the recording’s embedding model. Tenant A retires a model version and its answers leave in one wave: 433 removed, and 2 under a retention hold kept out of service; when a handbook page changes for Tenant B, 252 answers stay in Probation, marked as possibly out of date. Every number in a caption was read off the page as it was recorded. The file is the one archived on Zenodo.

Grafferene is a response cache for LLM endpoints. In measure-only mode it sits in front of an OpenAI-compatible endpoint for a week, every request still reaches the model, and the report shows what caching would have saved and the answers it would have served, side by side, for the owner to judge.

When serving is on, an answer comes from the cache only if every check passes: an exact repeat in the same tenant and namespace, or a reworded question asked in the same context as the original, optionally confirmed by a separate verification service. If verification cannot complete, the match is not served. Everything else goes to the model.

Measured on one node. Against vLLM on a laptop GPU, measure-only mode reported exactly the hits a serving run of the same 32-request trace produced, row by row, while all 32 requests reached the model. On that trace, built to repeat, a strict threshold served 13 of 32 and cut prompt tokens by 41% with no wrong answers on the trace; the same question under different instructions, or in another namespace, always reached the model. Over one million real embeddings (laptop CPU, search only), the approximate index found every match exhaustive search found, at 10 ms per search instead of 87 ms. Savings were checked against the model server’s own counters.

Stated on the site rather than left to be discovered: in public open chat only about 3% of prompts repeat across users, and on human-labeled look-alike question pairs about one similar-question match in five at the default threshold is a different question. That is why the report shows the answers, not only the rate.

Transferable principle: measure what a change would do on real traffic before letting it act, and show the cases it would get wrong.

References

Zhao et al. (2024). WildChat: 1M ChatGPT Interaction Logs in the Wild. ICLR 2024. arXiv:2405.01470
Wang et al. (2019). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. ICLR 2019 (source of the QQP question pairs). arXiv:1804.07461
Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023, 611–626. doi:10.1145/3600006.3613165

grafferene.ca demo.grafferene.ca Sample report Demo video: Zenodo 10.5281/zenodo.23329121 Grafferene on LinkedIn