Re-architecting an AI agent to reduce token cost, latency, and context overhead
A production resume agent had grown to 99 context files and 38,924 provider input tokens per request, driving latency and cost beyond a sustainable runtime. I redesigned the system around persistent hybrid retrieval, task-aware context budgets, prompt caching, and durable execution — reducing warm-cache known cost by 77.13%, context assembly by 98.22%, and HTTP lifetime by 99.29% while preserving 100% requirement coverage.
Use fewer tokens without losing decision quality
The challenge
A production resume agent was repeatedly sending a growing body of context into each request: 99 files and 38,924 provider input tokens. That made the system slower and more expensive, eventually pushing the request lifecycle beyond 90 seconds. The real problem was not simply model speed — it was unnecessary context and an inefficient request architecture.
My role
I owned the problem framing, system/product decisions, implementation direction, validation, and iteration represented in this case study. The emphasis is on the decisions that changed the system—not a feature inventory.
Architecture follows the constraint
Key decisions
- Treat timeout extension as measurement accommodation, not the architectural fix.
- Move retrieval work out of the synchronous request path and into persistent hybrid retrieval.
- Use task-aware context budgets to reduce unnecessary provider input instead of loading broadly “just in case.”
- Gate optimization on quality: faster and cheaper was unacceptable if requirement coverage regressed.
Working loop
The implementation was treated as a measured loop: diagnose the actual constraint, make the smallest architectural change that resolves it, then verify behavior and quality before moving forward.
What I built
- Persistent hybrid retrieval indexes durable evidence before request time instead of reparsing 99 files on every request.
- Task-aware context budgets assemble only the information required for the current job, reducing unnecessary token input.
- Prompt caching reuses stable provider context instead of paying repeatedly for the same input.
- Durable async execution separates long-running work from HTTP request lifetime.
- Provider-authoritative token audits track actual model usage, while requirement-level quality gates prevent cost savings from degrading output quality.
Validation in practice
Validation
- Context assembly reduced 98.22%.
- HTTP request lifetime reduced 99.29%.
- Request-time file parsing reduced from 99 files to 0.
- Warm-cache known cost reduced 77.13%.
- 100% requirement coverage and 100% Critical Fact Recall preserved.
What changed
The system stopped treating “more context” as automatically better. Retrieval, budgeting, and caching reduced repeated token usage and runtime overhead, while quality gates preserved the evidence contract and decision quality. The result was a substantially cheaper and faster AI runtime, combining AI runtime optimization with lower token usage without trading away requirement coverage.
Core capabilities
This case sits inside a broader portfolio spanning AI systems and agents, full-stack products and revenue infrastructure, design engineering, and creative AI leadership.
Explore the full system of work
Each project focuses on a distinct product, engineering, or creative challenge. The current project is highlighted so the portfolio reads as one connected body of work rather than a collection of isolated case studies.