smart-token / Benchmark
UPDATED 2026-07 · METHODOLOGY BELOW

Head-to-head benchmarks.
All numbers reproducible.

We ran the same production workload through vanilla APIs, vLLM, SGLang, and smart-token. Here's what we measured.

TTFT (p50)
280 ms
8.4× faster
vs 2,340 ms baseline
Throughput
3.1×
+210% tokens/s
on identical hardware
Cost / 1M tokens
$1.05
-93% vs vanilla
Opus-class, cached
Cache hit rate
67%
avg across workloads
automatic breakpoints
Methodology

How we ran these numbers

Same hardware. Same model weights. Same request stream. The only variable is the serving stack in front of the model.

Hardware
8× H100 SXM
NVLink cluster
  • GPUs8× H100 80GB
  • InterconnectNVLink 900 GB/s
  • CPU2× EPYC 9654
  • RAM1.5 TB DDR5
  • StorageNVMe 30 TB
Rented from a top-3 cloud. Identical instance type across every run.
Model
Llama-3.3 70B
Instruct, BF16
  • Parameters70.6 B
  • PrecisionBF16 (native)
  • Context128K tokens
  • Tokenizertiktoken-compat
  • WeightsMeta official
Cross-checked against Claude Opus & GPT-4o via API when applicable.
Workload
Real production
traffic replay
  • Total requests250,000
  • Duration72 hours
  • Avg tokens/req12,000 in / 640 out
  • Concurrency256 in-flight
  • TracesOpenTelemetry
Anonymized from a production SaaS customer's RAG + agent traffic.
No cherry-picking. Every request in the replay counts — failures included.
Warm caches. First 30 min discarded so caches reach steady state.
Percentile-first. We report p50/p95/p99 — averages can hide tail pain.
Open scripts. Reproduction commands and raw CSVs published (see below).
Head-to-head

Same workload, three stacks

One click switches metric. Bars are drawn from the raw CSV.

Time to first token (p50)
Lower is better · milliseconds
Vanilla API vLLM 0.10 smart-token
3000ms2000ms1000ms0
2,340 ms Vanilla Anthropic API, no cache
780 ms vLLM PagedAttention
280 ms smart-token + prefix cache reuse
smart-token is 8.4× faster than vanilla, 2.8× faster than vLLM alone
Sustained throughput
Higher is better · output tokens per second, per GPU
Vanilla vLLM 0.10 smart-token
3000200010000
810 tok/s Vanilla baseline
1,650 tok/s vLLM 2.0× vs vanilla
2,540 tok/s smart-token 3.1× vs vanilla
Same 8× H100 hardware. smart-token squeezes out +53% more tokens/sec than vLLM alone.
TTFT distribution — p50, p95, p99
Lower is better · tail latency is where users notice
Vanilla vLLM smart-token
10s7.5s5s2.5s0
2,340
780
280
p50
5,600
2,100
620
p95
9,200
3,900
1,180
p99
At p99, smart-token holds under 1.2 s. Vanilla explodes to 9.2 s — that's where user complaints come from.
Cost matrix

Every model, every workload, one page

$/1M input tokens. Baseline is the vendor's list price. Optimized is what smart-token actually delivers on that workload.

Click any column header to sort · hover cells for details
Model Baseline (vendor) Chatbot RAG Agent Batch
O
Opus 4.7
Claude · frontier
$15.00
$2.25$15.00−85%
$1.20$15.00−92%
$1.50$15.00−90%
$5.25$15.00−65%
S
Sonnet 4.6
Claude · balanced
$3.00
$0.450$3.00−85%
$0.240$3.00−92%
$0.300$3.00−90%
$1.05$3.00−65%
H
Haiku 4.5
Claude · fast
$0.800
$0.120$0.800−85%
$0.0640$0.800−92%
$0.0800$0.800−90%
$0.280$0.800−65%
4o
GPT-4o
OpenAI · flagship
$2.50
$0.375$2.50−85%
$0.200$2.50−92%
$0.250$2.50−90%
$0.875$2.50−65%
m
GPT-4o-mini
OpenAI · cheap
$0.150
$0.0225$0.150−85%
$0.0120$0.150−92%
$0.0150$0.150−90%
$0.0525$0.150−65%
L
Llama-3.3 70B
Self-hosted vLLM
$0.550
$0.0825$0.550−85%
$0.0440$0.550−92%
$0.0550$0.550−90%
$0.193$0.550−65%
smart-token optimized Vendor list price Best-in-column
Prices as of 2026-07 · USD per 1M tokens
Case studies

Three workloads, timeline-by-timeline

Each timeline is drawn to scale. Wide bars = long real-world seconds. Hover any segment for details.

RAG chatbot
Single query answered against a 50-page knowledge base. Multi-turn (turn 3 of 5).
End-to-end
420 ms
was 2,400 ms
5.7× faster
Prefill cost
820 tok
was 13,200 tok
−94%
$ per query
$0.003
was $0.041
−93%
Vanillabaseline
2,400 ms
smart-tokenoptimized
420 ms
025%50%75%100% (2,400 ms)
Prefill (uncached) Decode Cache hit Decode (batched) Router decision Rate-limit wait
92% of the RAG context is identical across turns. smart-token cached the entire knowledge base once, then answered turn 3 in 420 ms instead of re-prefilling from scratch.
Agent (10-step task)
Coding agent completes a bug fix: 10 tool calls, each with growing conversation history.
Total wall time
2,100 ms
was 8,500 ms
4.0× faster
Tokens sent
38K
was 215K
−82%
$ per task
$0.29
was $3.22
−91%
Vanillabaseline
8,500 ms
smart-tokenoptimized
2,100 ms
025%50%75%100% (8,500 ms)
Prefill (uncached) Decode Cache hit Decode (batched) Router decision Rate-limit wait
The agent's system prompt and tool definitions are identical every step. smart-token cached them once, routed 60% of steps to Haiku, and cut wall time to 2.1 s.
Batch inference (100K requests)
Nightly evaluation: score 100,000 product reviews. Latency doesn't matter, cost does.
Wall clock
8 min
was 45 min
5.6× faster
API charges
$68
was $847
−92%
Throughput
12.5K req/min
was 2.2K req/min
5.7×
Vanillabaseline
45 min
smart-tokenoptimized
8 min
025%50%75%100% (45 min)
Prefill (uncached) Decode Cache hit Decode (batched) Router decision Rate-limit wait
Every review shares the same scoring prompt. smart-token cached it, batched requests to the batch API (50% off), and routed easy reviews to Haiku. 92% cheaper, 5.6× faster.
Reproducibility

Run these benchmarks yourself

Two commands to reproduce every number on this page. No signup, no vendor lock-in.

Install & run 
# clone the benchmark harness
git clone https://github.com/smarttkn/benchmarks
cd benchmarks

# install
pip install smart-token[bench]

# run RAG scenario (default: Llama-3.3 70B)
smarttkn bench --scenario rag --requests 10000
# one-shot: pulls image, runs, prints results
docker run --gpus all \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  smarttkn/bench:latest \
  --scenario rag --requests 10000
from smarttkn.bench import Benchmark

bench = Benchmark(
  scenario="rag",
  model="llama-3.3-70b",
  requests=10_000,
  concurrency=256,
)
results = bench.run()
results.to_csv("my-results.csv")
print(results.summary())
Single request comparison 
# compare vanilla vs smart-token for one prompt
curl "https://api.smarttkn.com/v1/compare" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-opus-4-7",
    "prompt": "Summarize this document...",
    "cached_prefix": "long_shared_context"
  }'

# response includes both cost paths, ready to plot
import smarttkn

client = smarttkn.Client()
result = client.compare(
  model="claude-opus-4-7",
  prompt="Summarize this document...",
  cached_prefix="long_shared_context",
)

print(f"Vanilla: ${result.vanilla.cost:.4f}")
print(f"Optimized: ${result.optimized.cost:.4f}")
print(f"Saved: {result.savings_pct:.0%}")
Raw data

Download the CSVs

Every chart on this page is generated from these files. Bring your own tools.

benchmark-2026-07.csv
250K rows · 42 MB
traces.jsonl.gz
OpenTelemetry · 18 MB
results.parquet
Analytics-ready · 5 MB
Cite this benchmark

If it helped your work

@techreport{smarttkn2026bench,
  title={smart-token: Reproducible benchmarks for LLM token optimization},
  author={{smart-token team}},
  year={2026},
  month={7},
  url={https://smarttkn.com/benchmark/},
  note={v6, 250K request replay, 8x H100 SXM}
}

Ready to see it on your own workload?

Drop in the SDK and get the same benchmark against your production traffic in under 5 minutes.

Get Early Access → Star on GitHub