Does Speaking to Agents Like Cavemen Really Save 65% of Tokens? We Test

JetBrains AI
Supercharge your tools with AI-powered features inside many JetBrains products
Does Speaking to Agents Like Cavemen Really Save 65% of Tokens? We Test
A paired A/B benchmark of the token-compression skill Caveman on Claude Code, run on SkillsBench: does it actually save tokens, and does it degrade AI agent output quality?Output-token saving on real agentic tasks, with the skill forcibly activated. This is the ceiling, not the usual-case result.
Why we ran this
We at JetBrains are investing more and more into proper testing of the tooling around coding agents, and one skill got our attention: “Caveman”. Its pitch is best described in its own dialect:
Skill make agent talk like caveman. Why use many token when few do trick. Filler die. Code, commands stay byte-exact. 65% output token saved. Every reply. Forever. Work with 30+ agents. Many GitHub star.
We think:
Claim cheap to make. Verify expensive. Agent not chat window. Agent output mostly tool call, file edit, code: skill promise not touch those. So we measure two things README not measure: real saving on multi-step agent work, and whether squeezing agent think-out-loud hurt task outcome.
Setup
| Harness | Harbor 0.17: Docker-sandboxed trials, task-level verifiers, paired runs. |
| Agent | Claude Code 2.1.200, headless, bypassPermissions . |
| Model | claude-sonnet-5 , reasoning effort low (--effort low ). |
| Benchmark | SkillsBench (benchflow/skillsbench ): 86 of 87 tasks. Each task is auto-graded by its own tests on a 0-1 scale, where 1 means solved and partial credit is possible. |
| Arm A | no-skill : stock Claude Code. |
| Arm B | with-skill-forced : Caveman installed via Harbor --skill plus one instruction line forcing activation: “Use caveman mode…” |
| Pairing | Same tasks, same model, same settings, same budget per arm; excluded tasks excluded from both arms. |
| Volume | 3 runs, about 240 billed trials, about USD 106 total. |
Why “forced” matters: Caveman is user-activated. It triggers on phrases like “caveman mode” or “be brief”. We forced it on in every reply, which means every number below is the skill’s best case. In normal use, where the agent must decide to activate it on its own, the realized saving can only be equal or lower than the roughly 10% ceiling measured here.
Finding 1: the saving is about 8.5%, not 65%
Advertised savings come from chat-style prose answers. Agentic output is different: code, diffs, tool invocations, and exact error strings dominate the token stream, and Caveman correctly leaves all of it verbatim. Only the narration between tool calls gets compressed, and there is not much of it.
Finding 2: no detectable quality degradation
The question we actually cared about: does making the agent terse make it worse? Across 82 paired tasks in the full run, the answer is no: the arms are statistically indistinguishable.
Style transfer itself works exactly as designed: forced-arm transcripts are unmistakably caveman, while code artifacts stay untouched and normal.
Finding 3: the cost saving is real but fragile
Cost tracks the roughly 8.5% token saving, so the skill arm should come out roughly 10% cheaper, and per task, it does. But the raw arm totals in our full run showed the skill arm 11.6% more expensive: USD 40.60 vs. USD 36.39. The entire inversion came from a single trial: one dependency-audit task ballooned past the 200k long-context pricing tier in the skill arm and billed USD 8.29 vs. USD 0.33. In an earlier run the same task threw a USD 3.25 outlier in the baseline arm. It is a property of the task, not the skill.
Outcome
Safe, honest about style, oversold on savings. Forced on, Caveman reliably changes how the agent talks without any measurable damage to what the agent produces: 82 paired tasks, sign test p = 0.82. But on real agentic work it trims about 8.5% of output tokens and about 10% of cost at absolute best, because the tokens that dominate agent sessions are code and tool calls, which the skill deliberately preserves. The advertised 65% belongs to chat-style Q&A, not to coding agents.
Recommendation: use it if you like it. It is fun, and it costs you nothing measurable in quality. Just do not expect huge savings on daily agentic tasks: a high-single-digit percentage is the realistic ceiling.
- Quality: no detectable degradation: 8 tasks better, 10 worse, 64 tied; average task score differs by 0.015 on a 0-1 scale (p = 0.82).
- Tokens: -8.5% output tokens with activation forced, meaning this is the ceiling; auto-triggered usage saves less or nothing.
- Cost: roughly -10% in expectation, routinely erased by single-trial variance.
- Methodology bonus: our first 10-task run “showed” a -30% token saving. It dissolved as sample size grew. Never trust a k=1 eval.
You want next skill tested? Drop name in comments. Few word enough. We test.
Run details: Harbor 0.17; claude-sonnet-5
with reasoning effort low; SkillsBench 86/87 tasks; about 240 trials; about USD 106 total spend.

Facts Only

The experiment involved a paired A/B benchmark of the token-compression skill "Caveman" on Claude Code run on SkillsBench across 86 out of 87 tasks. The setup used Claude Sonnet 5 with low reasoning effort. Arm A utilized the stock model, and Arm B included the Caveman skill forced via an instruction line. The testing involved 3 runs, totaling about 240 billed trials and USD 106 in total spend. Finding 1 indicated a saving of about 8.5% in output tokens for agentic tasks. Finding 2 showed no detectable quality degradation, as the arms were statistically indistinguishable across 82 paired tasks with a sign test p = 0.82. Cost tracking showed an expectation of approximately -10% cost saving, but real-world results included one outlier trial that caused variance in pricing for a specific task.

Executive Summary

Testing the token compression skill "Caveman" on Claude Code in a benchmark setting revealed that it achieves an approximate 8.5% reduction in output tokens for agentic tasks, which is significantly lower than the advertised 65%. This saving stems from the nature of agent output, where code and tool calls dominate the token stream, which the skill successfully preserves verbatim, while only compressing narrative elements between tool calls. The study also found no measurable degradation in output quality; the forced style transfer did not negatively affect task outcomes, as both skill-activated and non-activated arms were statistically indistinguishable across paired tasks.
The cost savings projected from this token reduction are complicated by variance observed in real-world billing; while the theoretical saving suggests a roughly 10% cost reduction, a single high-cost trial introduced significant outliers that obscured the overall trend. Therefore, while the skill is effective at achieving marginal savings on agentic workflows without quality impact, users should expect only modest, fragile cost benefits rather than substantial token savings on complex tasks.

Full Take

The discrepancy between the advertised 65% token saving and the measured 8.5% outcome highlights a critical gap between chat-style prose compression and complex agentic reasoning involving code generation and tool invocation. The finding that Caveman preserves crucial code artifacts while compressing only interstitial narration suggests that token efficiency in this domain is highly context-dependent; what appears as savings in conversational text is negligible when dealing with the atomic units of agent output like file edits or error strings. This pattern demonstrates a systemic issue where marketing oversimplifies performance metrics derived from casual interactions, failing to account for the distinct structure of functional programming outputs. The fragility of cost saving, shown by the single-trial variance, cautions against relying on aggregate efficiency numbers in production environments where task complexity varies. Future testing must focus not just on token count but on defining operational unit relevance—does compressing instruction flow equate to real user value when performance hinges on preserving specific artifacts?

Sentinel — Human

Confidence

This text appears to be a human-authored post presenting the results of a specific A/B testing experiment, characterized by technical detail and cautious interpretation of statistical outcomes.

Signals Detected

Sentence length variance is noticeable; shifts from formal reporting to direct argumentative assertion.

Strong internal logic connecting experimental setup, findings, and caveats, indicative of a specific analytical workflow.

Structured presentation of methodology (Setup, Finding 1, 2, 3, Outcome) mirroring scientific reporting patterns.

Use of highly specific, non-obvious statistical claims and internal conflict regarding cost tracking suggests primary human analysis layered over data.

Human Indicators

Presence of self-referential critique on methodology ('Never trust a k=1 eval'), specific reference to proprietary internal setup (Harbor, SkillsBench), and acknowledgment of real-world variance in pricing.

The tone balances promotional claims with rigorous, almost overly detailed statistical caveats, typical of expert technical writing.