AI coding assistants are powerful but only as good as their understanding of your codebase. When we pointed AI agents at one of Meta’s large-scale data processing pipelines – spanning four repositories, three languages, and over 4,100 files – we quickly found that they weren’t making useful edits quickly enough.
We fixed this by building a pre-compute engine: a swarm of 50+ specialized AI agents that systematically read every file and produced 59 concise context files encoding tribal knowledge that previously lived only in engineers’ heads. The result: AI agents now have structured navigation guides for 100% of our code modules (up from 5%, covering all 4,100+ files across three repositories). We also documented 50+ “non-obvious patterns,” or underlying design choices and relationships not immediately apparent from the code, and preliminary tests show 40% fewer AI agent tool calls per task. The system works with most leading models because the knowledge layer is model-agnostic.
The system also maintains itself. Every few weeks, automated jobs periodically validate file paths, detect coverage gaps, re-run quality critics, and auto-fix stale references. The AI isn’t a consumer of this infrastructure, it’s the engine that runs it.
The Problem: AI Tools Without a Map
Our pipeline is config-as-code: Python configurations, C++ services, and Hack automation scripts working together across multiple repositories. A single data field onboarding touches configuration registries, routing logic, DAG composition, validation rules, C++ code generation, and automation scripts – six subsystems that must stay in sync.
We had already built AI-powered systems for operational tasks, scanning dashboards, pattern-matching against historical incidents, and suggesting mitigations. But when we tried to extend it to development tasks, it fell apart. The AI had no map. It didn’t know that two configuration modes use different field names for the same operation (swap them and you get silent wrong output), or that dozens of “deprecated” enum values must never be removed because serialization compatibility depends on them.
Without this context, agents would guess, explore, guess again and often produce code that compiled but was subtly wrong.
The Approach: Teach the Agents Before They Explore
We used a large-context-window model and task orchestration to structure the work in phases:
- Two explorer agents mapped the codebase,
- 11 module analysts read every file and answered five key questions,
- Two writers generated context files, and
- 10+ critic passes ran three rounds of independent quality review,
- Four fixers applied corrections,
- Eight upgraders refined the routing layer,
- Three prompt testers validated 55+ queries across five personas,
- Four gap-fillers covered remaining directories, and
- Three final critics ran integration tests – 50+ specialized tasks orchestrated in a single session.
The five questions each analyst answered per module:
- What does this module configure?
- What are the common modification patterns?
- What are the non-obvious patterns that cause build failures?
- What are the cross-module dependencies?
- What tribal knowledge is buried in code comments?
Question five was where the deepest learnings emerged. We found 50+ non-obvious patterns like hidden intermediate naming conventions where one pipeline stage outputs a temporary field name that a downstream stage renames (reference the wrong one and code generation silently fails), or append-only identifier rules where removing a “deprecated” value breaks backward compatibility. None of this had been written down before.
What We Built: A Compass, Not An Encyclopedia
Each context file follows what we call “compass, not encyclopedia” principle – 25–35 lines (~1,000 tokens) with four sections:
- Quick Commands (copy-paste operations).
- Key Files (the 3–5 files you actually need).
- Non-Obvious patterns.
- See Also (cross-references).
No fluff, every line earns its place. All 59 files together consume less than 0.1% of a modern model’s context window.
On top of this, we built an orchestration layer that auto-routes engineers to the right tool based on natural language. Type, “Is the pipeline healthy?” and it scans dashboards and matches against 85+ historical incident patterns. Type, “Add a new data field” and it generates the configuration with multi-phase validation. Engineers describe their problem; the system figures out the rest.
The system self-refreshes every few weeks, validating file paths, identifying coverage gaps, re-running critic agents, and auto-fixing issues. Context that decays is worse than no context at all.
Beyond individual contextual files, we generated a cross-repo dependency index and data flow maps showing how changes propagate across repositories. This turns “What depends on X?” from a multi-file exploration (~6000 tokens) into a single graph lookup (~200 tokens) – in config-as-code where one field change ripples across six-subsystems.
Results
| Metric | Before | After |
| AI context coverage | ~5% (5 files) | 100% (59 files) |
| Codebase files with AI navigation | ~50 | 4,100+ |
| Tribal knowledge documented | 0 | 50+ non-obvious patterns |
| Tested prompts (core pass rate) | 0 | 55+ (100%) |
In preliminary tests on six tasks against our pipeline, agents with pre-computed context used roughly 40% fewer tool calls and tokens per task. Complex workflow guidance that previously required ~two days of research and consulting with engineers now completes in ~30 minutes.
Quality was non-negotiable: three rounds of independent critic agents improved scores from 3.65 to 4.20 out of 5.0, and all referenced file paths were verified with zero hallucinations.
Challenging the Conventional Wisdom on AI Context Files
Recent academic research found that AI-generated context files actually decreased agent success rates on well-known open-source Python repositories. This finding deserves serious consideration but it has a limitation: It was evaluated on codebases like Django and matplotlib that models already “know” from pretraining. In that scenario, context files are redundant noise.
Our codebase is the opposite: proprietary config-as-code with tribal knowledge that exists nowhere in any model’s training data. Three design decisions help us avoid the pitfalls the research identified: files are concise (~1,000 tokens, not encyclopedic summaries), opt-in (loaded only when relevant, not always-on), and quality-gated (multi-round critic review plus automated self-upgrade).
The strongest argument: Without context, agents burn 15–25 tool calls exploring, miss naming patterns, and produce subtly incorrect code. The cost of not providing context is measurably higher.
How to Apply This to Your Codebase
This approach isn’t specific to our pipeline. Any team with a large, proprietary codebase can benefit:
- Identify your tribal knowledge gaps. Where do AI agents fail most? The answer is usually domain-specific conventions and cross-module dependencies that aren’t documented anywhere.
- Use the “five questions” framework. Have agents (or engineers) answer: what does it do, how do you modify it, what breaks, what depends on it, and what’s undocumented?
- Follow “compass, not encyclopedia.“ Keep context files to 25–35 lines. Actionable navigation beats exhaustive documentation.
- Build quality gates. Use independent critic agents to score and improve generated context. Don’t trust unreviewed AI output.
- Automate freshness. Context that goes stale causes more harm than no context. Build periodic validation and self-repair.
What’s Next
We are expanding context coverage to additional pipelines across Meta’s data infrastructure and exploring tighter integration between context files and code generation workflows. We’re also investigating whether the automated refresh mechanism can detect not just stale context but emerging patterns and new tribal knowledge forming in recent code reviews and commits.
This approach turned undocumented tribal knowledge into structured, AI-readable context and one that compounds with every task that follows.
Facts Only
Meta developed a pre-compute engine using over 50 specialized AI agents.
The system analyzed a data processing pipeline spanning four repositories, three languages, and over 4,100 files.
59 concise context files were generated to encode tribal knowledge.
AI agent navigation coverage improved from 5% to 100% of the codebase.
50+ non-obvious patterns were documented, including hidden dependencies and serialization rules.
Preliminary tests showed a 40% reduction in AI agent tool calls per task.
Complex workflows now complete in ~30 minutes instead of two days.
The system is self-maintaining, with automated validation and updates every few weeks.
Context files follow a "compass, not encyclopedia" principle, averaging 25–35 lines each.
Multi-round critic reviews and quality gates were implemented to ensure accuracy.
A cross-repo dependency index and data flow maps were created to track change propagation.
The approach is designed for proprietary codebases with undocumented tribal knowledge.
Meta plans to expand the system to additional pipelines and integrate it with code generation workflows.
Executive Summary
Meta developed a pre-compute engine using over 50 specialized AI agents to systematically analyze and document a large-scale data processing pipeline spanning four repositories, three languages, and over 4,100 files. The system generated 59 concise context files encoding previously undocumented tribal knowledge, improving AI agent navigation from 5% to 100% coverage. These context files follow a "compass, not encyclopedia" principle, providing actionable insights in 25–35 lines per file. The system also documented 50+ non-obvious patterns, such as hidden dependencies and serialization rules, which were critical for maintaining code integrity. Preliminary tests showed a 40% reduction in AI agent tool calls per task, with complex workflows now completing in 30 minutes instead of two days. The infrastructure is self-maintaining, with automated jobs validating file paths, detecting coverage gaps, and auto-fixing stale references every few weeks. This approach contrasts with academic findings that AI-generated context files can decrease success rates in well-known open-source projects, as Meta's proprietary codebase contained unique, undocumented knowledge not present in model training data.
The system's design emphasizes conciseness, quality control through multi-round critic reviews, and automated freshness checks to prevent context decay. It also includes a cross-repo dependency index and data flow maps to streamline change propagation analysis. Meta plans to expand this approach to additional pipelines and explore tighter integration with code generation workflows. The framework is presented as adaptable for any large, proprietary codebase, with a focus on identifying tribal knowledge gaps, using structured questioning, and maintaining rigorous quality gates. The initiative highlights the challenges of applying AI to complex, proprietary systems where contextual understanding is critical for accurate and efficient development tasks.
Full Take
This initiative by Meta presents a compelling case for how AI can be harnessed to document and navigate complex, proprietary codebases—an area where traditional AI tools often struggle due to lack of contextual understanding. The strongest aspect of this narrative is its focus on addressing a real-world problem: the gap between AI capabilities and the tribal knowledge embedded in large-scale, proprietary systems. By systematically extracting and structuring this knowledge, Meta has not only improved AI efficiency but also created a self-sustaining infrastructure that reduces the cognitive load on engineers. The "compass, not encyclopedia" principle is particularly noteworthy, as it prioritizes actionable insights over exhaustive documentation, aligning with how engineers actually work.
However, the narrative also raises questions about scalability and generalizability. While the system works well for Meta’s specific pipeline, the article acknowledges that similar approaches failed in open-source projects like Django, where models already had sufficient pretraining data. This suggests that the success of such systems may be highly context-dependent, relying on the uniqueness of the codebase and the depth of undocumented knowledge. Additionally, the emphasis on automation and self-maintenance, while efficient, could introduce new risks if the critic agents or validation processes fail to catch subtle errors or evolving patterns.
The root cause driving this narrative is the tension between AI’s potential and its limitations in understanding human-created systems. The assumption here is that tribal knowledge is the primary bottleneck, and that structuring it for AI consumption will unlock productivity gains. This echoes historical patterns in software engineering, where tools like IDEs and static analyzers were developed to bridge the gap between human intuition and machine execution. The implications for human agency are mixed: while engineers may spend less time on repetitive tasks, the system also centralizes knowledge in a way that could make human expertise less visible or valued over time.
Bridge questions to consider: How might this approach change the role of senior engineers, who traditionally hold tribal knowledge? What happens if the AI-generated context files introduce new biases or blind spots? Would this system work as effectively in a smaller organization with fewer resources to maintain it?
Counterstrike scan: If this were part of a coordinated influence campaign, the playbook might involve overhyping AI’s ability to replace human expertise while downplaying the risks of automation failures. However, the article’s transparency about limitations (e.g., the contrast with open-source projects) and its focus on practical, measurable outcomes suggest a genuine effort to solve a real problem rather than push a manipulative narrative. The content does not align with a hypothetical attack pattern.
Patterns detected: none
Sentinel — Human
The text reads like a highly technical case study written by an experienced engineer detailing a complex system build, demonstrating high levels of human authorship and expertise.