This article is the second of two parts on adaptive parsing, in Part III of Enterprise Document Intelligence, a series that builds an enterprise RAG system from four bricks: document parsing, question parsing, retrieval, and generation. The first part, Loop engineering with adaptive PDF parsing: start cheap, pay for a heavier parser only when the page needs it (link to come), built the escalation cascade and the deterministic checks that flag a failed parse for free. This part adds the LLM as the last line of defence and walks two real escalations end to end.

📓 The runnable companion walks both escalations end to end: you parse the flat-table page with fitz

then re-parse it with Azure, send the figure page to a vision LLM, and watch context_structured

flip the generation from a wrong answer to a right one. On GitHub: doc-intel/notebooks-vol1.

PyMuPDF parses a page in five milliseconds for free. A vision LLM on the same page can cost ten thousand times more and take ten seconds. When the question lives in plain prose, the cheap parser wins. When the answer hides inside a table, a figure, a scanned region, or a flattened layout, the cheap parser silently returns nothing useful and the LLM confidently answers the wrong question.

Running the heaviest parser on every page is wasteful. Running the cheapest one everywhere is wrong. The trick is to start cheap and escalate only when something says the cheap parse missed the answer. That signal has to come from the pipeline itself: a chain of checks against the parse output, the question, and the LLM’s own footprint when it tries to use what came out.

The way around this is a feedback loop the pipeline builds for itself. Start with the cheapest parser. Evaluate its output at every check of the pipeline. Escalate to a deeper parser only when at least one check flags the cheap parse as insufficient for the question.

The evaluation is not one check at one place. It is a cascade:

• pre-parsing metadata routes most of the corpus before any extraction
• parsing-time outputs flag flattened tables and opaque figures
• retrieval scoring catches anchors that drift
• generation flags what survived all of the above

Each check is cheaper than the next, more reliable than the next, and asks its own question: “did the parser produce enough to answer?”.

The first part caught failures cheaply, before generation. This part is what happens when they slip through: the LLM’s own footprint at generation time (check 7, self-evaluation; check 8, groundedness) flags a parse the deterministic checks missed, and the pipeline escalates the offending page to a deeper parser. Two recurring examples are walked end to end: Table 3 of the Attention paper (Vaswani et al. 2017, under the arXiv non-exclusive distribution license) (page 9, a flat-parsed grid escalated to Azure Layout) and Figure 1 (page 3, an opaque diagram escalated to a vision LLM). Both are real, both come up in enterprise corpora every day.

1. The generation-side checks: check 7 (self-eval) and check 8 (groundedness)

Generation is the last stop in the cascade. Its two checks are check 7 (the LLM’s self-evaluation flag, baked into the typed answer) and check 8 (a separate-LLM or NLI groundedness check after the answer is produced). Check 7 is the most expensive AND least stable check in the panorama, so this part spends real time on it: two walkthroughs that show it firing usefully (Case A in section 2, Case B in section 3), then a stress test that says the LLM signal alone is not enough (section 4). Check 8 is section 5.

Before the two walkthroughs: most questions do not need any escalation at all. Article 9 ran the standard pipeline on the same Attention paper with the question “What are the options for positional encoding?” and got a clean answer at the first generation pass, context_structured=True

, no re-parse needed. PyMuPDF alone was enough because the answer lived in prose. Same for “What are the six CSF Functions?” on the NIST document. The two walkthroughs below are the harder cases, where PyMuPDF could not see what the question needed.

Two questions on the Attention paper, two failure shapes, two parser escalations. The walkthroughs mirror each other on purpose: same data model, different parser at the deep step.

2. Case A: flat-table escalation, end to end

The question: “What is the value of h for the base model in Table of Variations on the Transformer architecture?” The correct answer is 8

, readable from the h

column of the base

row of Table 3 on page 9 of the Attention paper. The cell is filled (not blank) so the answer itself is unambiguous; what stresses the pipeline is the flat-parsed shape of the surrounding grid, which is exactly what the check-2 fingerprint already flagged for free in the first part and what the LLM signal at check 7 also flags below.

2.1. Question, used directly

Article 6 builds a question-parsing brick (parse_question

) that handles typos, multi-part questions, language detection, and expert-vocabulary expansion. On a question this short and clean, the brick is overkill. We skip it and feed the question text directly to retrieval and generation. The pipeline does not break; we just choose not to call one of the four bricks because the input is already in the shape downstream wants.

The keywords we pick by eye for the retrieval call below: just the table-caption string, “Table 3: Variations on the Transformer architecture”. One specific phrase is enough on this corpus.

2.2. Parsing: PyMuPDF, with bounding boxes

The standard pipeline starts cheap with PyMuPDF. One function, one line:

line_df = fitz_pdf_to_line_df("data/paper/1706.03762v7.pdf")

page_df = build_page_df(line_df)

line_df.shape ~ (1500, 8)

columns : page_num, line_num, text, x0, y0, x1, y1, character_count

The first eight lines of the captured line_df

(page 1, before we ever look at page 9):

2.3. Retrieval: which method finds page 9, and how

Article 7 covers four retrieval methods: keyword matching, embedding similarity, TOC matching, and an LLM-orchestrator on top. For this question, keyword retrieval is sufficient because the question mentions a specific string (“Table 3”) that the table caption echoes verbatim on page 9. We pass that exact string as the only keyword:

from docintel.retrieval import retrieve_pages

retrieved_pages_df, filtered_line_df = retrieve_pages(

page_df, line_df,

keywords=["Table 3: Variations on the Transformer architecture"],

top_k=5,

)

The captured retrieved_pages_df

:

The filtered_line_df

returned alongside retrieved_pages_df

carries every line of page 9, ready for the generation brick.

2.4. First generation: gpt-4.1 on PyMuPDF, the LLM flags the parse

The generation brick (llm_answer_with_evidence

, Article 8) takes the filtered lines and the question, calls gpt-4.1 with a Pydantic schema as the output contract. The AnswerWithEvidence

schema already carries context_structured

(added in Article 8), so the orchestrator has a one-bit trigger to read on every answer:

from docintel.generation import llm_answer_with_evidence

from docintel.generation.qa.schemas import AnswerWithEvidence

pass1: AnswerWithEvidence = llm_answer_with_evidence(

question=("What is the value of h for the base model "

"in Table of Variations on the Transformer architecture?"),

filtered_line_df=fitz_p9, # PyMuPDF lines for page 9

)

AnswerWithEvidence carries :

answer, start/end_page_num, start/end_line_num, confidence,

justification, caveats,

context_structured <-- the binary trigger for the cascade

complete_answer_found, llm_discovered_keywords

The captured response, copy-pasted from the real run:

{

"answer": "8",

"start_page_num": 9,

"start_line_num": 25,

"end_page_num": 9,

"end_line_num": 25,

"confidence": 0.99,

"justification": "In the table listing parameters for different Transformer ...",

"caveats": [],

"complete_answer_found": true,

"context_structured": false,

"llm_discovered_keywords": []

}

The answer is "8"

, the correct value. The citation points to PyMuPDF L25 of page 9. Look back at the parse: L25 IS the h

cell of the base row (base

is at L21, then values L22=6 (N), L23=512 (dmodel), L24=2048 (dff), L25=8 (h)). The LLM got both the right value AND the right cell.

But context_structured=False

. The LLM flags the structural risk. PyMuPDF returned the 13 base-row cells as 13 separate lines L22-L33, the column headers on lines L5-L20 (some single-line, some split across two lines for compound headers like “train steps”). To map L25 to “h” the LLM had to count column positions in a flattened header. It got it right this time, but it warns: the structure is fragile, do not trust the next similar question without checking.

The binary flag is the trigger. When False

, the orchestrator escalates regardless of how confident the answer was.

2.5. Pipeline escalates: re-parse page 9 with Azure DI

context_structured=False

. The orchestrator picks a structure-aware parser. The project has Azure DI wired up, but Camelot, Docling, or any other table-aware parser would slot in identically:

from docintel.parsing.pdf.azure_layout import azure_layout_pdf_to_line_df

azure_p9 = azure_layout_pdf_to_line_df(

"data/paper/1706.03762v7.pdf",

pages=[9], # only the page retrieval pointed at

)

azure_p9.shape ~ (52, 9)

columns : page_num, line_num, text, x0, y0, x1, y1, character_count, parsing_method

every row carries parsing_method == "azure_layout"

The first eight Azure rows on page 9:

The text

column holds Azure’s markdown, one line per table row. Decoded back into a grid, that is Table 3 as Azure recovered it, the structure PyMuPDF had flattened:

2.6. Second generation: same model, structurally trusted this time

Same call as Pass 1, with the Azure rows instead of the PyMuPDF rows. The captured response:

{

"answer": "8",

"start_page_num": 9,

"start_line_num": 7,

"end_page_num": 9,

"end_line_num": 7,

"confidence": 0.98,

"justification": "In Table 3 (line 7), the 'base' model lists its hyperpara...",

"caveats": [],

"context_structured": true

}

Same value, this time the LLM trusts the structure. Reading the Azure row, the model does not need to count columns: h

is the 5th cell of the markdown line, the value 8

is right there, the column header h

is the 5th cell of L5 above. Citation is the whole row instead of one cell, but the cell-to-column mapping is unambiguous because the pipes anchor everything.

The feedback loop fired and resolved. Cost: one extra Azure DI call (about $0.003 and four seconds). Gain: the answer is now structurally trusted, and line_df

carries a permanently structured page-9 record for any future question on this page.

2.7. Annotation: highlight the cited cell on the PDF

The annotation brick converts the evidence span into a rectangle drawn on the source PDF. We use the Pass 1 citation (PyMuPDF L25, the h

cell of the base row) because PyMuPDF returns per-cell lines with tight bboxes; the Azure row would give a row-wide box covering all 13 cells:

from docintel.rendering.pdf.annotation import (

passage_lines_df_from_answer,

passage_bbox_by_page,

draw_passage_rectangles,

)

passage_df = passage_lines_df_from_answer(line_df, pass1)

bboxes_df = passage_bbox_by_page(passage_df)

draw_passage_rectangles(

pdf_path="data/paper/1706.03762v7.pdf",

bboxes_df=bboxes_df,

out_pdf_path="output/parsing_comparison/h_value_annotated.pdf",

)

The captured passage_df

is one row:

And bboxes_df

, the same row collapsed per page:

The rendered annotated PDF page (top half of page 9, cropped to Table 3):

End-to-end auditability: the question travelled from text input through four bricks and back to a rectangle on the source PDF. A reviewer can scan the highlighted page and verify in one second that the cited cell matches the LLM’s claim. Even though the feedback loop fired (Pass 1 cs=False -> escalate), the loop did not break the audit trail; it added an Azure-parsed row for the same page, kept the PyMuPDF rows alongside, and the annotation used the PyMuPDF bbox because that is what produced the tightest box.

2.8. Sidebar: same question across three models and two parsers

The walkthrough above used gpt-4.1. To see how parser quality interacts with model strength on this exact question, we ran the same call across three models (gpt-4o-mini, gpt-4o, gpt-4.1) and both parsers (PyMuPDF, Azure DI). The grid below should return "8"

in every cell; the interesting variation is the context_structured

flag:

Two findings.

First, the answer is the same everywhere. This question is unambiguous on the source: the value 8

sits in a normal filled cell, the LLM picks it up reliably regardless of parser or model.

Second, the context_structured

flag varies in a useful way. gpt-4o-mini and gpt-4o accept PyMuPDF’s flat lines without flagging. gpt-4.1 conservatively flags the structural risk, even though it got the answer right. The pipeline’s reliability depends on which model populates the flag: a stronger model is a more honest reporter of upstream parsing risk. With Azure DI, every model trusts the structure; the parser closes the model-strength gap on the flag side.

The takeaway: upgrading the parser is the durable fix, because it removes the structural risk altogether. Upgrading the model raises the chance of detecting parsing risk, but does not remove it. Production pipelines that mix both knobs hedge against either weakness.

3. Case B: figure escalation, end to end

The question: “What is the architecture of the Transformer?” The answer is mostly carried by Figure 1 on page 3 (the encoder-decoder architecture diagram). The prose around it describes pieces of the architecture but does not replace the diagram.

Pass 1, PyMuPDF only: Retrieval finds page 3. PyMuPDF returns the surrounding prose plus a type='image'

row for Figure 1, with a placeholder text and no actual content. line_df

around the figure looks roughly like:

Generation, pass 1: The LLM reads the prose, notices the placeholder and the gap, and produces a partial answer with a structural caveat:

{

"answer": "The Transformer uses a stack of 6 encoder layers and 6 decoder l...",

"start_page_num": 3,

"start_line_num": 12,

"end_page_num": 3,

"end_line_num": 14,

"confidence": 0.55,

"caveats": [

"Figure 1 is referenced in the prose but its content is absent fr..."

],

"complete_answer_found": false,

"context_structured": true

}

Pipeline calls vision LLM on the image. Azure DI would not help here: the diagram is not a table, it is free-form vector geometry. The pipeline reads the caveat “figure referenced but absent”, looks up image_df.image_id=1

, sees parsing_method='not_parsed'

, and routes to a vision-language model:

new_lines = vision_extract(

"data/paper/1706.03762v7.pdf",

pages=[3],

image_ids=[1],

)

new_lines.parsing_method == "vision_gpt4o"

image_df : row image_id=1 updated from parsing_method='not_parsed' to 'vision_gpt4o'

The vision model returns a structured description: encoder input embedding plus positional encoding, six identical layers each carrying multi-head self-attention with residual connection and layer norm, then position-wise feed-forward with residual and norm; decoder mirror with masked self-attention plus cross-attention to encoder output; final linear and softmax to vocabulary. This description gets appended to line_df

as new text rows with parsing_method='vision_gpt4o'

.

Generation, pass 2: The LLM now has both the prose description and the structured visual description. It produces a complete answer with high confidence, naming the residual connections, the layer norm placement, the cross-attention bridge: details the prose alone did not enumerate.

{

"answer": "Encoder-decoder. Encoder is 6 identical layers ; each layer is m...",

"start_page_num": 3,

"start_line_num": 30,

"end_page_num": 3,

"end_line_num": 42,

"confidence": 0.92,

"complete_answer_found": true,

"context_structured": true

}

The full cost: 5 seconds PyMuPDF, about 10 to 30 seconds and a few cents for the vision call on one image. The other 14 pages of the paper never see a vision model.

4. Stress tests on the LLM signal (what 18 runs revealed)

The walkthroughs in sections 2 and 3 show the LLM signal firing usefully on two real questions. That is the best case. To see what the signal looks like outside the best case, we stress-tested it across questions of growing structural difficulty, three models, and both parsers. The picture is less reassuring than a single happy-path walkthrough suggests, and it justifies the cascade order: let the cheap deterministic checks at checks 1 and 2 carry the load; keep the LLM signal as the last line of defence, not the first.

4.1. The stress matrix: 18 cells, four wrong answers, zero canaries

We ran three questions of growing structural difficulty against Table 3 of the Attention paper, on three models (gpt-4o-mini, gpt-4o, gpt-4.1) and two parsers (PyMuPDF, Azure DI). Eighteen runs in total.

The questions:

• Q_easy: the value of

h

for thebase

row. Answer is8

, sitting in a directly-labelled cell. - Q_inheritance: the value of

h

for the rows labelled(B)

. Answer is8

again, inherited frombase

because theh

cell is blank for(B)

. The flat parse loses the column anchor. - Q_far: the perplexity of the

big

model on the dev set. Answer is4.33

, last row of a long table with many similar-looking numbers around.

The starting framing was: a weaker model that cannot answer can still flag the broken parse as a structural canary. The data says the opposite happens. When a weak model gives the wrong answer on Q_inheritance

, it does not flag. It fabricates and asserts context_structured=True

.

Out of four wrong answers across the matrix, four had cs=True

and zero had cs=False

. Three different model-parser pairs on Q_inheritance

all picked a number from an adjacent column (dk

= 16 or 32) and asserted the structure was clean. On Q_far

, gpt-4o-mini on PyMuPDF returned 25.7 (the BLEU column, not PPL) with cs=True

. The model that mis-aligned columns also failed to notice it had mis-aligned them.

The single cs=False

in the matrix came from gpt-4.1 on PyMuPDF for Q_inheritance

. Its answer was 8

, correct. Read at face value: the only time the LLM signal fired, it was a model that did not need to be rescued.

4.2. The continuous score does not help either

The natural next move is to replace the binary flag with a 0.0-1.0 score, expecting the LLM to use the middle of the range when uncertain. We re-ran the matrix with a context_structured_score

field added.

Scores clumped between 0.85 and 1.00 regardless of correctness. Mean score on correct answers: 0.919. Mean score on wrong answers: 0.920. Identical. At threshold 0.95, the score catches all three wrong answers, but raises nine false alarms on the fifteen correct ones, a 60% false-alarm rate. No usable threshold.

The LLM’s structural verdict is bimodal disguised as continuous. The score buys nothing beyond the binary flag.

4.3. The LLM can describe the parse, but its verdict is the unstable part

Asking gpt-4.1 for a one-sentence rationale alongside the flag changes the picture. The rationale text is concrete and consistent across runs: “headers split across multiple lines”, “each cell on its own line, not aligned”, “blank cells indicate inherited values”. The model accurately describes the parse shape every time.

The verdict on top of the rationale, on the other hand, is unstable. We ran gpt-4.1 three times on Q_inheritance

with PyMuPDF, same input, same temperature settings:

The model knows what is wrong with the parse. It does not know reliably whether that is enough to escalate. A feedback loop that reads the rationale and applies a deterministic rule on top of it (“contains ‘split across multiple lines’ or ‘each cell on its own line’ or ‘blank cells’” then escalate) becomes stable. Reading the binary or the score directly does not.

5. Check 8: post-generation groundedness

After the answer is produced, a second LLM (or an NLI model) verifies that every claim in the answer is supported by the cited chunks. This catches the fabrications the generating model is too confident to flag itself in check 7. Cost: one extra LLM call. Reliability: higher than the same model self-evaluating its own answer, because the judge is fresh and has no incentive to defend the generation. Cheaper and more reliable than the check-7 LLM-as-judge approach for parsing quality, and the only check in generation that has a chance of catching fabrications check 7 missed.

The cascade as a whole says: checks 1 and 2 fire on every page in milliseconds (document parsing); check 4 fires on the question (question parsing); checks 5 and 6 fire after retrieval (retrieval); check 7 only fires if the upstream checks gave the page a clean bill of health (this section); check 8 fires after generation as a final safety net (this section). The walkthroughs above cover what check 7 looks like when it is the last line of defence; the panorama says that should not be the only line of defence.

6. Operational notes

Two practical refinements before closing: when the lazy default is wrong, and how the data model gives caching for free.

6.1. When eager parsing wins anyway

Lazy parsing is the right default. It is not the right call everywhere:

• Permanent, heavily-queried documents. A standard insurance policy template is queried thousands of times across a portfolio’s lifetime. Parse it once with Docling or Azure at ingestion, store the result, amortize over all future queries.
• Audit-grade documents: Due diligence, regulatory filings, litigation. When you need a defensible record that no information was missed, full parsing upfront is the baseline. The cost is acceptable because the alternative (missing a relevant clause) is far worse.
• Small documents: Under 20 pages, the marginal cost of running Docling or Azure on the whole thing is small. The 15-page Attention paper might fall in this bucket: the cost-benefit shifts.
• Pre-computed search indexes: Full-text retrieval over a corpus parses once at ingestion. Lazy parsing applies within a document at query time, not at corpus ingestion time.

The right question to ask: how many times will this document be queried, on average, across its lifetime? Many times, go eager. A few times or unknown, go lazy. For a typical enterprise corpus where most documents are queried zero or one times, lazy parsing wins by a wide margin.

6.2. The data model IS the cache

When the pipeline re-parses page 6 of the Attention paper with Azure, the result lands in line_df

with parsing_method='azure_layout'

and in page_df

with the corresponding audit row. The next question that touches page 6 hits these rows directly. No re-parsing needed. The cache is not a separate layer; it is the same line_df

and page_df

that retrieval already reads.

def get_or_enrich(pdf_path, line_df, page_df, pages, method):

"""Re-use a previous enrichment if one already ran with the same method."""

done = page_df[

(page_df.page_num.isin(pages)) &

(page_df.parsing_method == method)

]

missing = [p for p in pages if p not in done.page_num.values]

if not missing:

return line_df, page_df # already enriched

return enrich(pdf_path, line_df, page_df, missing, method)

Over time, the cache fills with the enrichments that questions triggered. The most-queried regions of the most-queried documents get the deep-parsed version automatically. The rarely-touched regions stay on the cheap parse. The system learns where to invest deep parsing based on actual usage, not on a guess made at ingestion time.

7. Conclusion

Parsing quality is a multi-check decision spread across the four bricks, ordered cheap-first: deterministic signals during parsing (char density, flat-table fingerprint, chunk integrity), then intent-aware routing during question parsing, then score-gap and context-gap checks during retrieval, then LLM self-eval as the last line of defence. The stress test in section 4 makes the ordering non-negotiable: the LLM’s binary verdict on its own context is not stable enough to lead the cascade, and a separate fresh judge is what catches the fabrications.

The data model carries the cascade: parsing_method

as a column on every parsing table makes mixed parses, escalation, audit, and caching reduce to filtering on a column. The same skeleton supports any new failure shape: adding a new shape adds a row to the routing map, adding a new check adds a column. Article 11 keeps the same self-correction frame and applies it to cross-references.

8. Sources and further reading

The reference for what advanced parsing does and its per-page cost is Docling (Auer et al., Docling Technical Report, 2024). Table extraction as a separate problem from text extraction is grounded by Smock et al. (PubTables-1M / Table Transformer, CVPR 2022). The vision-LLM-per-page cost tier (~5-30 seconds on a GPU) is from Blecher et al. (Nougat, Meta 2023). The LLM-as-feedback-signal pattern the article uses to drive escalation is in the same family as Asai et al. (Self-RAG, ICLR 2024). The article reports a concrete negative result from an 18-run stress test: LLM self-evaluation as the binary parser-quality verdict is unreliable on its own; a separate-LLM groundedness check on the produced answer catches what the in-loop signal misses.

Earlier in the series:

• Document Intelligence: series intro. What the series builds, brick by brick, and in what order.

What works, what breaks

• Baseline Enterprise RAG, from PDF to highlighted answer. The four-brick pipeline end to end: PDF in, highlighted answer out.
• Embeddings Aren’t Magic: The Predictable Failure Modes of RAG Retrieval. Where embedding similarity wins (synonyms, typos, paraphrase), where it predictably breaks (unknown terms, negation, term-vs-answer relevance), and how to use it anyway.
• Rerankers Aren’t Magic Either: When the Cross-Encoder Layer Is Worth the Cost. What a cross-encoder adds over bi-encoder embeddings, measured, and when it is worth the latency.
• RAG is not machine learning, and the ML toolkit solves the wrong problem. Why chunk-size sweeps and finetuning optimize the wrong thing; route by question type instead.
• From regex to vision models: which RAG technique fits which problem. Two axes, document complexity and question control, that pick the technique for each case.
• 10 common RAG mistakes we keep seeing in production. Ten production mistakes, organized brick by brick, with the fix for each.

Document parsing

• Beyond extract_text: the two layers of a PDF that drive RAG quality. The first half of the parsing brick: the document’s nature, signals, and summary.
• Stop returning flat text from a PDF: the relational tables RAG needs. The second half of the parsing brick: the relational tables every downstream brick reads.
• Parse scanned PDFs for RAG with EasyOCR: free OCR gives you words, not a document. Where traditional OCR stops: text recovered, structure lost.

Question parsing

• RAG questions need parsing too: turn the user’s string into briefs for retrieval and generation. The thesis of question parsing: why a user string needs the same parsing as a document, and how it splits into a retrieval brief and a generation brief.
• What the question parser extracts from a user string: keywords, scope, shape, decomposition, clarification. The five families of columns the parser reads straight from the user’s question, with the code that fills each one.
• Dispatching the parsed RAG question: chunk strategy, model tier, activations, audit. The decisions the parser makes on top of the user string, using the document’s profile: dispatch, activations, full schema, the audit trail (pipeline_trace.json), and a broker-corpus walkthrough.
• The Clarification Loop and Learned Defaults: When the Question Is Not Precise Enough. One focused clarification when the question is too vague, and the default learned from the answer.

Retrieval

• Retrieval is filtering, not search: a mental model for enterprise RAG. Retrieval reframed as filtering on line_df and toc_df: anchors small, context large.
• Anchor detection for RAG: parallel detectors, then one LLM call at the end. Parallel anchor detectors: keyword always, embeddings alongside, one LLM call at the end.
• Letting an LLM pick the right RAG page: the arbiter pattern at the end of retrieval. The LLM arbiter: candidates ranked with reasons, one typed JSON out.
• Context Engineering: The Four Typed Inputs Behind Every Answer. Context engineering given a structure: the four typed pieces (fixed system prompt, retrieved lines, doc-context block, PromptContext wrapper) that fill one single-document RAG LLM call.

Generation

• Make RAG generation return a typed contract: citations, typed values, and self-checks (link to come). The answer schema as the contract: typed values, items with evidence spans, self-assessment fields, and the completeness signal the pipeline computes itself.
• Assemble each RAG generation prompt from a base prompt plus the rules each question needs (link to come). The dispatcher: a fixed BASE prompt plus the rules each question needs, the schema picked from the registry, and the full trace kept on every call.
• Validating the RAG answer before the user sees it: spans, quotes, and the feedback loop (link to come). The post-generation validator (spans, verbatim quotes, formats), not-found as a first-class answer, and the feedback loops that close the pipeline.

One-document pipelines

• A production RAG pipeline for PDFs: relational parsing, TOC retrieval, typed answers (link to come). Each of the four bricks upgraded one contract at a time: relational parsing, corpus-aware questions, TOC-routed retrieval, typed answers.
• Stop RAG hallucinations with context engineering: one pipeline, four very different PDFs (link to come). The four upgraded bricks wired into one call, run end to end on a paper, a compliance doc, and a broken-TOC document.
• Loop engineering with adaptive PDF parsing: start cheap, pay for a heavier parser only when the page needs it (link to come). The escalation cascade and the free deterministic checks that flag a failed parse before you pay for a deeper one.