Lean Thinking for AI Teams

Lean manufacturing and LLM engineering look unrelated. One optimizes physical flow of parts; the other optimizes probabilistic text generation. But both are production systems with high variability, and lean thinking has seventy years of answers for exactly that.

Waste hides in prompts, not just in code

The seven classic wastes map surprisingly well onto AI product work. Overproduction is generating far more tokens than the decision needs. Waiting is the idle time between a model call and a human review that never gets scheduled. Defects are hallucinations discovered downstream¹.

The most dangerous waste, though, is overprocessing: asking a large model to do in one giant prompt what a small pipeline could do deterministically.

# Overprocessing: one mega-prompt doing everything
answer = llm(f"Read all {len(docs)} documents and give me the final decision")

# Leaner: small batches, deterministic where possible
relevant = retrieve(docs, query, k=8)          # deterministic retrieval
claims = [extract_claim(d) for d in relevant]  # small, testable units
decision = adjudicate(claims)                   # model only where judgment is needed

The second version is worse at looking impressive and better at everything else: each stage is measurable, debuggable, and cheap to re-run.

Small batches beat big prompts

In manufacturing, large batches hide defects until thousands of units are made. In LLM systems, “large batches” are long contexts: put 200 pages into one prompt and errors hide in the middle, notoriously hard to detect².

The lean countermeasure is the same in both worlds:

1. Reduce batch size — fewer documents per call, shorter chains.
2. Build quality in — validate at each station, not at the end.
3. Stop the line — make it cheap to halt on anomalies instead of propagating them.

For a compact quantitative example, let inline flow efficiency be $E = T_v / T_l$, where $T_v$ is value-adding time and $T_l$ is total lead time. The same relationship can be shown as a block:

$$E = \frac{T_v}{T_l}, \qquad 0 \leq E \leq 1$$

Flow efficiency over resource efficiency

Teams often optimize for “model utilization” — keeping the expensive model busy. Lean would call that resource efficiency, and warn that it destroys flow efficiency: the speed at which one unit of value (one correct answer, one resolved ticket) moves through the system.

A model that is 100% busy but produces answers that wait three days for human verification is a bottleneck factory with a very expensive machine.

What I take from this

• Treat prompts and pipelines as production lines: instrument them, don’t admire them.
• Prefer many small, boring, testable steps over one clever mega-prompt.
• Measure the time from “question asked” to “decision made”, not tokens per second.

For the knowledge-systems side of this argument, see the companion piece on LLM 时代的知识工程.

Footnotes

1. This is an example footnote using standard Markdown syntax — rendered natively without any plugin. ↩

2. The “lost in the middle” phenomenon is well documented in long-context retrieval literature; treat this sample citation as a placeholder. ↩