Everyone talks about training LLMs. The cost, the data, the compute. But here's what most people miss: once your model is trained, the hard part is just starting. Serving that model to real users, fast, reliably, and without burning through GPUs, is where the actual engineering challenge lives. And it turns out, the biggest bottleneck isn't compute. It's memory.

I spent the past week studying the foundational papers behind vLLM, SGLang, and modern LLM serving infrastructure. What I found changed how I think about AI systems entirely.

Your GPU is mostly wasting memory

Take a 13B parameter model like Llama-2. Its weights alone consume about 26GB on an A100, that's already 65% of the GPU's 40GB memory. You'd think the remaining 35% gives you decent room to serve users. It doesn't.

The reason is the KV-cache. Every time an LLM generates a token, it needs to store key-value pairs from all previous tokens in the conversation. This cache is different for every request, grows dynamically as the response gets longer, and nobody knows in advance how big it will be. Legacy systems dealt with this by pre-allocating the maximum possible memory for each request, just in case.

The result? Measured waste of 60-80% of KV-cache memory. Not a small inefficiency. A fundamental architectural problem that limits how many users you can serve on the same hardware.

The three kinds of waste

When I dug into the vLLM paper, the memory problem became clearer through three distinct types of fragmentation. Internal fragmentation: slots are reserved but never filled because the response was shorter than the maximum. External fragmentation: gaps between allocated blocks that are too small to use but too scattered to reclaim. And reserved slots: memory locked for a request that hasn't even started generating yet.

Your slide deck from these papers shows this perfectly, the gap between "optimal memory" and "legacy static allocation" is enormous. Most of that gap is just wasted space that no request will ever touch.

PagedAttention: borrowing from the 1960s

This is the part that clicked for me. The vLLM team didn't invent a new neural architecture or a clever training trick. They looked at how operating systems solved the exact same problem fifty years ago.

In the 1960s, computers had the same issue with RAM: programs needed contiguous blocks, memory got fragmented, and the system ran out even when total free space was sufficient. The solution was virtual memory with paging, break memory into small fixed-size pages, let programs think they have contiguous space, but scatter the actual pages wherever there's room.

PagedAttention does this for the KV-cache. Instead of one monolithic slab per request, the cache gets split into small blocks (typically 16 tokens each). A block table maps logical blocks to physical locations in GPU memory, just like an OS page table. Blocks can live anywhere. When a request needs more space, allocate another block wherever there's room. When it finishes, free the blocks instantly.

The result: memory waste drops from 60-80% to under 4%. Same GPU, 2-4x more concurrent users.

Continuous batching: no more idle GPUs

PagedAttention solves the memory problem, but there's a second bottleneck: scheduling. With static batching, the GPU waits for every request in a batch to finish before starting new ones. If request A generates 10 tokens and request B generates 500, the GPU sits idle for 490 tokens worth of time on A's slot.

Continuous batching (first proposed in the Orca paper) fixes this by evaluating the queue after every single token generation step. The moment a request finishes, a new one takes its slot. No waiting, no idle cycles. Combined with PagedAttention's flexible memory, this means the GPU stays busy doing useful work nearly 100% of the time.

SGLang and RadixAttention: thinking in trees

vLLM's PagedAttention was the breakthrough, but SGLang pushed the idea further. Modern LLM workloads aren't just simple chat, they involve structured outputs (JSON schemas), multi-turn tool use, and many requests sharing the same system prompt. Standard engines treat each fork as a completely separate request, redundantly recomputing the shared prefix.

SGLang's RadixAttention manages the KV-cache as a radix tree (a prefix tree). If three users share the same system prompt, that prompt is computed once and stored once. Only the divergent branches, the unique parts of each conversation, consume new memory blocks. For agentic workloads where many agents share common context, this is a massive win.

What I learned

The thing that surprised me most is that none of the key breakthroughs in LLM serving came from machine learning research. PagedAttention came from operating systems. Continuous batching came from job scheduling theory. RadixAttention came from data structure design. The entire field of LLM inference optimization is really a systems engineering discipline wearing an AI hat.

This reframed how I think about my own PhD direction. I've been working as an AI engineer deploying LLMs in production, and I always thought the interesting research was in the models themselves. But the real bottleneck, the thing that determines whether Morocco's planned Jazari Institutes can actually serve AI to citizens, is the infrastructure layer. How you manage memory, schedule requests, and orchestrate multiple models on limited hardware.

That comparison table from the papers stays with me: there is no "one size fits all" memory allocator. The right approach depends on your hardware (HBM vs LPDDR5) and your workload (simple chat vs agentic multi-call programs). For a country building AI infrastructure from scratch, this means the serving stack needs to be designed for the specific hardware and use cases available, not just copied from hyperscaler playbooks.

That's exactly the gap my PhD thesis aims to fill.

Papers & Resources

  • Kwon et al. (2023), Efficient Memory Management for LLM Serving with PagedAttention, arxiv.org/abs/2309.06180
  • Zheng et al. (2024), SGLang: Efficient Execution of Structured Language Model Programs, arxiv.org/abs/2312.07104
  • Yu et al. (2022), Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022
  • Miao et al. (2025), Towards Efficient Generative LLM Serving: A Survey, dl.acm.org/doi/10.1145/3754448
  • Gordic (2025), Inside vLLM: Anatomy of a High-Throughput LLM Inference System, aleksagordic.com/blog/vllm

I'm Mustapha Liaichi, an AI engineer exploring the frontier of LLM systems and autonomous agents. These notes document my research journey. Reach me at mustaphaliaichi@gmail.com