Here's a number that changed how I think about LLM inference: modern LLMs spend roughly 90% of their time doing nothing. Not computing. Not reasoning. Just waiting for weights to travel from memory to the compute units. The GPU is sitting there, billions of transistors ready to go, starving for data.

This is the fundamental physics of autoregressive generation. And the two most elegant solutions I've found attack it from completely different angles — one plays with time, the other plays with space. Together, they represent what I'd call the paradigm of intelligent asymmetry.

The physics you can't negotiate with

LLM inference is bounded by two hard constraints that no amount of clever engineering can remove. Only work around.

The time problem: generating K tokens requires K serial forward passes through the model. One token, one pass. You can't parallelize this the way you parallelize training, because each token depends on the previous one. Arithmetic intensity during decode is capped at roughly 1 FLOP per byte in FP16. That's absurdly low for hardware designed to do trillions of operations per second.

The space problem: weight loading dominates everything. In the attention layers, weights are 79x larger than activations. In the feed-forward layers, that ratio jumps to 1700x. Your GPU's compute cores finish their math almost instantly, then sit idle while the next chunk of weights crawls over from HBM. Memory bandwidth — not compute — dictates your peak throughput.

Once you see these two constraints clearly, the two optimization strategies I studied this week become obvious in hindsight. One attacks the time constraint. The other attacks the space constraint.

Speculative decoding: not all tokens need a 70B brain

The key realization behind speculative decoding is almost funny in its simplicity: most tokens in a sentence are easy. The word "the" doesn't need 70 billion parameters to predict. Neither does "is" or "and" or the closing bracket in a JSON response.

So why run the full model for every single token?

Speculative decoding splits the work between two models: a tiny drafter (as small as 6M parameters) and the full target model. The drafter races ahead, generating γ candidate tokens cheaply. Then the target model evaluates all γ tokens in a single parallel forward pass — because verification is parallelizable even though generation isn't.

The drafter proposes. The verifier checks. Accepted tokens pass through. The first rejected token gets corrected by resampling from an adjusted distribution. And here's where it gets beautiful.

The math that guarantees zero degradation

This was the part that genuinely surprised me. Speculative decoding isn't an approximation. The output distribution is mathematically identical to running the full target model alone.

The verification works through a probability check: if the drafter's probability q(x) for a token is less than or equal to the target's probability p(x), the token is automatically accepted. If not, it's rejected with probability 1 - p(x)/q(x), and the system resamples from the adjusted distribution p'(x) = norm(max(0, p(x) - q(x))).

The result: P(output) = min(p,q) + (p - min(p,q)) = p(x). The target model's distribution, exactly. Not approximately. Not "close enough." Identical.

In practice, a T5-XXL target with a T5-Small drafter achieves 3.4x speedup. A GPT-like 97M target with a 6M drafter reaches an acceptance rate of 0.88 — meaning 88% of the small model's guesses are correct. That's 88% of tokens generated at near-zero cost.

The catch? Speculative decoding works best when memory bandwidth is the bottleneck and excess compute is available. In other words, exactly the situation most real deployments face.

The space problem: how do you compress 4x without lobotomizing the model?

A 70B model in FP16 needs 140GB just for its weights. An RTX 4090 has 24GB. A MacBook M1 has 64GB unified. The numbers don't work. You need to compress.

Naive 4-bit post-training quantization (RTN — round-to-nearest) fits the hardware. But it collapses the model's reasoning. Perplexity explodes. The model becomes a confident idiot — smaller, faster, and wrong.

The AWQ paper (Activation-Aware Weight Quantization) found something remarkable: the problem isn't that most weights are sensitive to quantization. It's that a tiny fraction — 0.1% to 1% — are disproportionately important. And you can't identify them by looking at the weights themselves.

Activations reveal what weights hide

This was the second insight that surprised me. The traditional approach to quantization looks at weight magnitudes — keep the big weights, quantize the small ones. It doesn't work well. AWQ instead looks at activation patterns during inference. The weights that correspond to high-activation channels are the salient ones, regardless of their magnitude.

Protecting just that 0.1-1% of activation-salient weights drops the perplexity collapse from 43.2 to 13.0. But you can't just keep those weights in FP16 and quantize the rest — mixed-precision formats are hardware-inefficient. The GPU has to constantly switch between different number formats.

AWQ's solution is elegant: before quantization, multiply each salient weight channel by a scaling factor s > 1, expanding its value range. Inversely scale the corresponding activation by 1/s to compensate. Now apply uniform INT4 quantization to everything. The artificially enlarged salient weights suffer less relative rounding error, while the math stays equivalent because the scaling factors cancel out.

Result: pure INT4, uniform format, full hardware efficiency. But with the accuracy of mixed-precision. A 70B model that couldn't fit on an RTX 4090 now runs at interactive speeds. A 13B model runs on a Jetson Orin Nano at 22.5 tokens/sec — 3.5x faster than FP16 baseline. A 7B model runs on a Raspberry Pi. Completely offline.

The comparison matrix

Studying AWQ alongside GPTQ and RTN clarified when to use what:

  • AWQ wins on hardware efficiency (pure INT4), calibration robustness (needs 10x less calibration data than GPTQ), and cross-domain generalization (perplexity shifts by only +0.5 when calibrated on PubMed and evaluated on Enron emails, versus +4.89 for GPTQ).
  • GPTQ is better when you have abundant calibration data and want to squeeze maximum accuracy through heavy weight reconstruction.
  • RTN is for when you need something quick and dirty and can tolerate quality loss.

The synthesis: intelligent asymmetry

What ties speculative decoding and AWQ together is a shared insight: not everything in an LLM is equally important, and exploiting that asymmetry is the key to efficient inference.

Speculative decoding exploits time asymmetry — not all compute steps are equal, so decouple cheap generation from expensive verification. AWQ exploits space asymmetry — not all parameters are equal, so protect the 1% that matters and compress everything else.

Both techniques recognize that treating the model as uniform (every token equally hard, every weight equally important) wastes resources. The smarter approach is to be asymmetric on purpose.

By shrinking the space requirement 4x through quantization, you shift the bottleneck from memory-bound back to compute-bound. Then speculative decoding exploits the freed-up compute to process multiple tokens per step. They're complementary — which is why modern serving stacks combine both.

What I learned

The thing that hit me hardest this week is how physical the constraints are. This isn't a software problem that better code can fix. It's a physics problem — memory bandwidth, arithmetic intensity, data movement costs. The solutions are clever because they work within the physics rather than pretending they can override it.

For anyone building AI systems on limited hardware — which is most of the world outside a handful of hyperscalers — these techniques aren't optional optimizations. They're the difference between "it technically runs" and "it actually works in production." A 70B model that needs 140GB is useless on real hardware. A 70B model quantized to 35GB with speculative decoding running at 3x speed is deployable. Same model. Same quality. Different engineering.

Papers & Resources

  • Leviathan et al. (2023), Fast Inference from Transformers via Speculative Decoding, arxiv.org/abs/2211.17192
  • Lin et al. (2024), AWQ: Activation-Aware Weight Quantization for LLM Compression and Acceleration, arxiv.org/abs/2306.00978
  • Dettmers et al. (2023), QLoRA: Efficient Finetuning of Quantized Language Models, arxiv.org/abs/2305.14314
  • Dao (2024), FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, arxiv.org/abs/2307.08691
  • Chandra (2026), On-Device LLMs: State of the Union, v-chandra.github.io/on-device-llms
  • Park et al. (2026), Accelerating Language Giants: A Survey of Optimization Strategies for LLM Inference on Hardware Platforms, FGCS

I'm Mustapha Liaichi, an AI engineer exploring the frontier of LLM systems and autonomous agents. These notes document my research journey. Reach me at mustaphaliaichi@gmail.com