Baseline · self-hosted
x86_64 EC2Qwen2.5-3B-Instruct · bf16, no quantization · transformers + PyTorch
How most people deploy today.
Cloud AI · Arm CPU inference
Two baselines, one optimized stack
Compare sends one prompt of yours to all three at once. Benchmark runs a fixed 50-prompt suite against the two boxes on AWS and reports the medians.
The two baselines are the two things you would realistically do instead. Self-hosted: Qwen2.5-3B the way most people deploy today — full precision Hugging Face + PyTorch on a stock x86_64 EC2 box, zero optimization. Hosted API: Gemma 4 26B-A4B with reasoning enabled, on someone else's GPU, reached over the network.
Against them: a Qwen2.5-0.5B/1.5B cascade on AWS Graviton4 — q4_k_m weights, llama.cpp built for Neoverse-V2, a router that keeps easy prompts on the small model, and speculative decoding when they escalate. Only the self-hosted baseline is a controlled comparison (matched vCPU and RAM, same weights family); the hosted API is a different model on different hardware, so read that ratio as a reference point rather than an isolated variable.
⌘/Ctrl + Enter to run · the self-hosted baseline runs at ~5 tok/s, so this costs it about 19 s
Try:
Baseline · hosted API
Someone else's GPUGemma 4 26B-A4B (MoE, 3.8B active) · reasoning on · OpenRouter free tier
What you get without owning any hardware.
Optimized cascade
AWS Graviton4 (aarch64)Qwen2.5-0.5B + 1.5B · q4_k_m · llama.cpp + KleidiAI (SVE2/i8mm/bf16)
Router + speculative decoding on Arm.
Run the suite
The same fixed prompts go to both AWS instances — the unoptimized x86 baseline and the Graviton4 cascade — one request at a time per box, both boxes at once. Nothing is cached or replayed: every number below is measured live in this run.
50 prompts × 64 tokens ≈ 11 min 21 s at worst — the baseline's ~5 tok/s sets the pace, and most prompts stop early on an end-of-text token. Requests to one box are spaced ~1.2 s apart to stay under its 1 req/s rate limit. The tab keeps running if you switch back to Compare.
What the 50 prompts are
Five groups of ten, interleaved so a 10- or 25-prompt run stays balanced instead of being all trivia. The 15 prompts from bench/bench.py are in here verbatim, so a browser run lines up with a CLI run.
- Factual10 prompts · should route to 0.5B
e.g. “What is the capital of France?”
- Everyday10 prompts · should route to 0.5B
e.g. “Hi there!”
- Reasoning10 prompts · should route to 1.5B
e.g. “Explain step by step how a B-tree index speeds up SQL queries.”
- Code10 prompts · should route to 1.5B
e.g. “Write a Python function that finds the longest palindromic substring.”
- Analysis10 prompts · should route to 1.5B
e.g. “Compare merge sort and quicksort, including their trade-offs in practice.”
Concurrent load — controlled single-instance benchmark
Real run — both EC2 hosts, 127.0.0.1, 20 req per level, max 96 tokens.What happens when N requests hit one box at the same time. The client ran on each EC2 host against 127.0.0.1, deliberately bypassing the public nginx 1 req/s/IP protection (which is untouched) — this measures server behaviour, not internet latency. Neither stack parallelizes requests: the baseline queues on an asyncio.Lock and llama.cpp runs -np 1 (one slot), so concurrency raises latency, not throughput. These are committed aggregate numbers, not a live run — raw rows: bench/concurrent_results.csv.
Throughput vs concurrency
requests/sec — higher is betterTail latency vs concurrency
p95 request latency — lower is betterBaseline (3B, x86)
- Peak throughput
- 0.07 req/s
- Peak aggregate generation
- 5.0 tok/s
- Saturation
- concurrency 2
- p95 latency at saturation
- 38.6 s
- Errors
- 0
Cascade (0.5/1.5B, Graviton4)
- Peak throughput
- 1.10 req/s
- Peak aggregate generation
- 74.7 tok/s
- Saturation
- concurrency 2
- p95 latency at saturation
- 4.35 s
- Errors
- 0
| System | Conc. | Req/s | tok/s | p50 lat | p95 lat | med TTFT | Err |
|---|---|---|---|---|---|---|---|
| Baseline | 1 | 0.07 | 5.0 | 19.2 s | 19.4 s | 491 ms | 0 |
| Baseline | 2 | 0.07 | 5.0 | 36.1 s | 38.6 s | 19.6 s | 0 |
| Baseline | 4 | 0.07 | 5.0 | 51.9 s | 76.5 s | 38.9 s | 0 |
| Baseline | 8 | 0.07 | 5.0 | 116.5 s | 137.3 s | 99.0 s | 0 |
| Baseline | 16 | 0.07 | 5.0 | 140.5 s | 246.5 s | 121.9 s | 0 |
| Cascade | 1 | 0.84 | 57.4 | 1.02 s | 2.16 s | 29 ms | 0 |
| Cascade | 2 | 0.92 | 62.5 | 1.53 s | 4.35 s | 548 ms | 0 |
| Cascade | 4 | 0.99 | 67.4 | 2.59 s | 9.19 s | 1.42 s | 0 |
| Cascade | 8 | 1.08 | 73.2 | 3.02 s | 13.2 s | 1.94 s | 0 |
| Cascade | 16 | 1.10 | 74.7 | 4.74 s | 16.1 s | 4.25 s | 0 |