Here's something I've been turning over since studying the Stoica & Zaharia paper this week: we are building a new kind of software, and we're doing it with tools designed for the old kind.
In traditional software engineering, you write explicit instructions. If x > 0, do this. Otherwise, do that. The program does exactly what you tell it, every time. When it breaks, it throws an exception. You read the stack trace. You fix the bug. The whole discipline — version control, unit tests, CI/CD, code review — grew up around this model of deterministic, line-by-line code.
LLM-based systems work nothing like this. You write a prompt in English. The model interprets it probabilistically. The same input can produce different outputs. When it fails, it doesn't crash — it confidently produces something wrong. There's no stack trace. There's no line number to fix. The weights that produced the bad output are a black box. You can't open them up and find the bug.
Stoica and Zaharia call this the shift from Software 1.0 to Software 2.0, building on Karpathy's original argument. In 1.0, programmers write every line of code. In 2.0, programmers describe a behavior they want, and optimization algorithms search for a program (a set of weights) that produces it. The source code isn't Python anymore. It's datasets. It's prompts. It's specifications of intent.
The Ford Model T problem
The analogy that stuck with me most is the comparison to the early automotive industry. Before the Model T, cars were bespoke. Every vehicle was hand-built. Parts from one Model K wouldn't fit another Model K. There was no standardization, no interchangeable components, no assembly line.
That's where LLM systems are right now. Every agent pipeline, every RAG system, every multi-model workflow is hand-wired. Components from one project don't plug into another. There's no equivalent of a standardized bolt or a universal interface. The research is producing increasingly powerful individual models, but the engineering infrastructure to compose them into reliable systems barely exists.
Stoica and Zaharia argue that the path forward requires specifications — precise descriptions of what a system should do and how to verify that it did it. They split this into two types: statement specifications (what should the task accomplish?) and solution specifications (how do we verify the output is correct?). In traditional engineering, these are your PRD and your unit tests. In AI engineering, the statement spec is often a prompt, and the solution spec is... well, that's the problem.
Who validates the validators?
This is the question I keep coming back to. In compound AI systems, you often use one LLM to evaluate the output of another LLM. It sounds reasonable until you think about it for more than a minute.
The evaluator LLM has the same biases, the same formatting sensitivities, the same tendency to hallucinate as the generator. You've built a feedback loop where a system grades itself. The HELM benchmark makes this concrete: when you rank models across seven dimensions (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency), the rankings shift dramatically depending on which dimension you prioritize. One model scored as the most toxic in the benchmark also turned out to be the least gender-biased. Another model that was the most accurate on standard tasks was the least accurate on irregular linguistic patterns. There's no single "best model." There's only "best model for this specific metric, measured this specific way."
That leads to a problem. If your evaluator is an LLM, and your evaluator has these same blind spots, you can't actually trust your evaluation pipeline unless you've validated it against human judgment. But human judgment is slow and expensive, and we're building systems faster than humans can grade them. That's the speed trap.
Criteria drift: you don't know what "good" means until you see "bad"
This one hit close to home. Criteria drift is the phenomenon where your definition of "good output" changes as you grade more examples. You start with a rubric. You grade ten outputs. By output fifteen, you realize your rubric was wrong — or at least incomplete. You refine it. But now your first ten grades are inconsistent with your updated criteria.
I've experienced this building AI applications at work. You deploy a system, the client says "it's not working right," and when you ask what "right" means, they can't articulate it until they show you an example of a bad output. The criteria emerge from the failures, not the other way around. This is the opposite of how traditional software testing works, where you define the test first and then build the code to pass it.
The researchers call this the catch-22 of AI evaluation: you need criteria to grade outputs, but you need to see outputs to define the criteria. Evaluation criteria aren't fixed constants. They're dependent variables that shift as you observe the system.
The bottleneck has moved
The last insight that changed my thinking is about where the constraint is in AI development now. When building software was slow and expensive, the bottleneck was writing code. Now that AI can generate code at near-zero marginal cost, the bottleneck has shifted. It's no longer about building — it's about knowing what to build.
Andrew Ng points out that development time has compressed so much that the ratio of engineers to product managers is trending from 7:1 down to 1:1 or even solo developers doing both. The hard part isn't the implementation. It's the product intuition. It's the ability to look at twenty prototypes and know which two are worth scaling. It's the user empathy to understand what someone actually needs versus what they asked for.
For an AI engineer, this means the skill stack looks different from what most people expect. Yes, you need to understand RAG architectures, agent workflows, and tool integration. But the differentiator isn't technical depth — it's evaluation discipline and product judgment. The teams that win are the ones that can systematically figure out why their system is failing, not the ones with the fanciest model.
What this means for my research
This notebook reframed how I think about my own thesis. I came into this PhD preparation focused on the infrastructure layer — how to serve LLMs efficiently, how to coordinate multi-agent systems. Those are real problems and they matter. But Stoica and Zaharia's paper made me realize that the layer above infrastructure is equally unsolved: how do you specify, evaluate, and debug these systems once they're running?
My thesis asks how to build resource-efficient multi-agent LLM systems. But efficiency isn't just about throughput and memory. A system that runs fast but can't be evaluated, debugged, or verified isn't useful in production. The infrastructure work (Notebooks 1-2) and the coordination work (Notebook 3) need to be paired with evaluation methodology that actually works for compound systems. That's the gap.
AI engineering as a discipline is maybe five years old. The tooling is where traditional software engineering was in the early 2000s — before standardized testing frameworks, before CI/CD pipelines, before git became universal. We're building increasingly powerful systems with increasingly primitive engineering practices. The people who figure out the evaluation and specification layer will shape the field as much as the people who build the models.
Papers & Resources
- Stoica & Zaharia et al. (2024) — AI Engineering: Building Applications with Foundation Models — arxiv.org/abs/2412.05299
- Liang et al. (2023) — Holistic Evaluation of Language Models (HELM) — arxiv.org/abs/2211.09110
- Shankar et al. (2024) — Who Validates the Validators? Aligning LLM-Assisted Evaluation with Human Preferences — arxiv.org/abs/2404.12272
- Karpathy (2017) — Software 2.0 — karpathy.medium.com
- Ng (2024-2025) — AI Engineering vs ML Engineering — DeepLearning.AI talks
I'm Mustapha Liaichi, an AI engineer exploring the frontier of LLM systems and autonomous agents. These notes document my research journey. Reach me at mustaphaliaichi@gmail.com