Overview
A client running AI-agent workflows on OpenClaw wanted to know whether a context-compression layer (“the Optimizer” below — internal naming, not the plugin’s own) would deliver real token and cost savings without quietly degrading output quality. Rather than take the vendor’s numbers on faith, this was tested directly: the same real workload, run twice — Optimizer off, then on — with every token, call, and dollar accounted for, and the actual output judged for quality both times, not assumed.
The result: −66% cost, −28% total tokens, −58% model calls, identical output quality (93.5/100 on both runs). Full raw evidence is linked inline throughout this write-up — both audit PDFs and the client report — so the numbers are checkable, not just asserted.
Read the full client report (PDF) →
What we tested
Workload: a real run of an AI-visibility audit skill that checks how ready a website is for AI crawlers and assistants (Google AI Overviews, ChatGPT, Perplexity, etc.) to find, understand, and cite it. Chosen as the test workload precisely because it’s a realistic, tool-heavy, multi-step job, not a synthetic benchmark.
Target: a real, independent SaaS marketing site — name withheld here, not the client’s own site — chosen as a realistic stand-in with a normal mix of pages, docs, and structured content. The audit PDFs linked throughout this page are the actual, unedited output and do name the site directly; publishing them alongside an anonymized write-up means anyone can re-run the same audit against the same public URL and check the numbers themselves. That’s the whole point of publishing raw evidence rather than a summary.
Model: OpenAI GPT-5 Mini via OpenRouter — a fast, low-cost model, reasoning mode off, identical settings on both runs.
Runs: 2 paired runs of the identical audit — Optimizer off, then on. Same prompt, same model, same target, same day.
Results
| Metric | Baseline (off) | Optimized (on) | Change |
|---|---|---|---|
| Model calls | 12 | 5 | −58.3% |
| Input tokens | 152,704 | 145,985 | −4.4% |
| Output tokens | 66,422 | 10,818 | −83.7% |
| Cache-read tokens | 548,736 | 206,080 | −62.4% |
| Total tokens billed (in+out) | 219,126 | 156,803 | −28.4% |
| Total tokens processed (incl. cache) | 767,862 | 362,883 | −52.7% |
| Cost | $0.1875 | $0.0643 | −65.7% |
| Target site’s AI Visibility Score | 93.5/100 | 93.5/100 | no change |
| Completed successfully | yes (1/1) | yes (1/1) | — |
Where the savings came from: almost entirely re-read context (cache) and generated output, not the input side. Cache-read volume dropped 62%, output tokens dropped 84% — the Optimizer compresses what the agent has to keep re-reading and re-writing on every step, not how much it “thinks” up front.
On the score staying constant: this is the rating the audit gave the target site — the thing that could have gotten worse if compression made the agent sloppier. It didn’t move at all between runs. (See both full audit PDFs linked above — same score, same six-pillar breakdown, same findings.)
The reliability journey
The clean result above wasn’t the first attempt. An earlier pass, on a heavier reasoning model, showed real compression (one page-fetch step alone dropped from ~300k tokens to ~20k) — but every repeat run with the Optimizer on failed to finish (0 of 6 completed), while the same task with it off finished every time.
Root-caused, not worked around: the failures weren’t the Optimizer’s fault. A platform-level timeout setting (agents.defaults.timeoutSeconds) was silently clamped to a hardcoded 120-second idle ceiling by a different code path than the one that setting appears to control — a regression of an already-closed upstream bug. The Optimizer’s extra processing time simply made runs more likely to cross that hidden ceiling. Filed upstream as openclaw/openclaw#129146, with the confirmed fix being to also set the provider-level timeout, which unblocks the ceiling through a different path. Re-tested clean on a second model afterward — the paired run above completed on the first try, both ways.
Where to introduce this first
The savings above came almost entirely from compressing large, disposable tool output sitting in context — not from the model thinking less. That points to a specific kind of workflow being the best fit, not “everywhere at once.”
Strong fit — pilot here first:
- Any step that fetches, parses, or evaluates a large external payload (a page, a document, an API response, another tool’s raw output)
- High call volume, where a per-call cost cut compounds
- Steps where only a judgment or a summary needs to survive past that step
Low priority — later, if at all:
- Short, low-volume conversational turns with little tool output to compress
- Latency-critical single-turn interactions, where the model has little large context to trim in the first place
- Anywhere the full raw history genuinely needs to persist for later steps
What this trial does and doesn’t prove
- One task type, one target site, one model — the exact percentages will move for a different workload; the direction (large tool-output steps benefit most) is the transferable finding.
- The target site’s score was re-checked, not independently fact-checked line-by-line beyond that.
- The reliability fix has been re-validated once, clean — worth a slightly larger sample before calling the reliability question fully closed.
- This used a stand-in target site, not the client’s own production workflow — where to roll it out on their real workload is a structural read from this trial’s shape, not yet a measurement on their traffic.
Full evidence: baseline audit (PDF) · optimized-run audit (PDF) · client report (PDF).