Case study: a research assistant you can audit
The business problem
Equity analysts, credit analysts and corporate finance teams spend a large share of their time on one kind of question: what did the company report, how has it changed, and what did management say about why. The answers sit in 10-K filings that run to a hundred pages or more, and the figures sit in financial statements in a different part of the same document.
Language models are now good at reading such documents and poor at being held to account for what they say about them. A model asked for Apple's operating margin will usually give a sensible number. It will not reliably say which year, which definition, which page, or whether it computed the figure or recalled it. For an analyst whose output goes to an investment committee or a client, an answer that cannot be checked has to be checked anyway, which removes most of the time it saved.
The useful question is therefore not "can a model answer questions about filings?" but "can a system answer them in a way a sceptical reader can verify in under a minute, and decline when it cannot?"
What was built
Footnote answers questions about the annual reports of fifteen large US companies across technology, consumer goods and industrials. It was built to four requirements:
- Every figure has a source a reader can open. Citations name the filing, the statement and the printed page, and link to the document on sec.gov.
- No figure is generated by a model. Reported figures are read from the companies' XBRL filings; changes, margins and compound growth are computed in code and recomputed by a verifier.
- Quotations are exact. When the answer is in the text rather than the numbers, the system quotes sentences word for word and the verifier confirms each one against its source passage.
- It says when it cannot answer. Questions about other companies, future results, currency conversions or investment decisions are refused with a reason. Questions that could mean two things ("profit") are answered with a question.
How a question is answered
A question is parsed for the companies, measures and years it mentions, then routed. A request for a figure becomes a database query; a request for a change adds a calculation; a comparison ranks companies on the same measure and notes when their fiscal years end in different months; a question about what a company said searches the text; a question about why a figure moved does both. The draft answer then passes through a verifier that checks every figure, citation, quotation, calculation, unit, period and stated direction of change. If a blocking check fails, the answer is withheld.
Why did Intel's operating margin decline in 2024?
Intel's operating margin fell from 0.2% in fiscal 2023 [1][2] to -22.0% in fiscal 2024 [3],
down 22.2 percentage points. In its fiscal 2024 annual report (Item 7, Management's Discussion
and Analysis), Intel attributes the change to the following:
- "Our consolidated gross margin in 2024 decreased by $4.4 billion, or 20%, compared to 2023 due
primarily to higher 2024 impairment charges and accelerated depreciation." [4]
- "To the extent we are unable to grow our revenues to offset these production costs, our gross
margin and operating income will be unfavorably affected." [5]
Results
On 72 hand-written questions across eight categories (figures, calculations, comparisons, explanations, document questions, ambiguous questions, questions outside the corpus, and questions designed to provoke a wrong answer), 71 were answered correctly. Routing was correct for all 72. Every question that should have been declined was declined, and nothing that should have been answered was declined. All 36 figures entered by hand from the published statements matched. Structured answers return in about 7 milliseconds; answers that search the text take about a third of a second. No paid model is called, so the marginal cost of a question is zero.
The evaluation report has the full breakdown.
What it gets wrong
The most instructive failure was found by hand, not by the evaluation. Asked to "compare Apple's 2024 revenue with Alphabet's 2023 revenue", the first version ranked both companies on 2024, dropped 2023 without saying so, and marked the answer verified, because every figure it stated was correct. The verifier was checking that what the answer said was supported; the fault was in what it left out. It is fixed, and it is the reason the failure analysis is as long as it is.
The one question still answered wrongly shows the same limit from the other side. Asked who Amazon names as competitors for AWS, the system quoted four exact, correctly cited sentences about AWS, none of which names a competitor. "Verified" in Footnote means the evidence is faithful. It does not yet mean the evidence is on point.
What this shows
- Reliability is a design property, not a model property. Most of the work is in the data model, the routing, the arithmetic and the checks, and very little is in a prompt.
- The boundary between what a model should do and what code should do can be drawn precisely. Here it is: code finds and computes every figure; a model, if used, may only rephrase sentences that already contain them, and its output is checked as strictly as anything else.
- Honest measurement includes the failures. The first run scored 95.5%, the faults were traced and fixed, the regression cases were added and labelled as such, and the remaining weakness is described rather than tuned away.
Try it
The hosted demo runs the complete system in your browser, with no server: the Python pipeline under Pyodide, the filings in an in-memory database, and the embedding model under transformers.js. It gives the same answers as the PostgreSQL deployment on every evaluation question.
Built with
Python, PostgreSQL with pgvector, SQLAlchemy, sqlglot, rank-bm25, sentence-transformers, LangGraph, FastAPI, Pydantic, structlog, Prometheus, pytest and Hypothesis, Docker, GitHub Actions, Pyodide and transformers.js for the browser build. Optional narration with the Anthropic API.