What it takes to turn counterfactual analysis into an AI product
One month ago your company decided to launch a brand new checkout process. The alternative hypothesis was very simple: a three-step checkout would drive more purchases and revenue than the previous one-step checkout, based on industry benchmarks and good UX/UI practices.
Just to make sure this decision really moves the needle (and with the aim of quantifying it), you suggested that an A/B test would be the best solution to measure the effect and reject the null hypothesis. The result: you couldn’t convince the stakeholders to run a properly designed A/B test and the decision of launching this new checkout was made anyway.
However, one month later, right now, those stakeholders ask questions such as: “what’s the effect of the new checkout?” “Did it work?” or “Did we lose money with it?”... I am very sure that most of you have been in this situation and might know some quasi-experimental techniques such as Difference-in-differences or Synthetic Control Methods but… what can you do when there are no other markets to compare against, and the company isn't big enough to build a credible synthetic control group?
I want to introduce ITSA (Interrupted Time Series Analysis), a method I’ve been using during the last four years in situations where you can’t run a randomized control trial and you have few options between all of the vast amount of techniques related to observational studies. Also, I will introduce how we created a multi-agent system to run ITSA, the architectural, statistical and software decisions (and trade-offs) we needed to make to develop this data product and the benefits this solution offers.
ITSA and why it matters
A brief introduction
Measurement in digital business has long trusted the pre- versus post-intervention approach for exactly these situations. I'm sure you've heard something like: "just compare what happened after the change with the period before." Maybe you've even believed it's a nice approach. Respectfully, let me show you why it falls short.
From a statistical point of view, comparing the mean before and after the intervention is a naive approach: it ignores pre-existing trends and seasonality, does not account for autocorrelation when quantifying uncertainty, and cannot distinguish the intervention from concurrent shocks. In the Figure 1 scenario, this comparison would lead you to conclude that the intervention increased daily orders by 10%. And you’d be wrong.
In Figure 2 you can see a different approach: instead of comparing the average value from pre and post-intervention, you can compare what happened post-intervention (Observed) with what would have happened if we hadn’t launched the three-step checkout (Counterfactual). In this case there is no positive or negative effect attributed to our intervention. Why? because the effect is measured by subtracting the counterfactual from the observed value.
How can you estimate the counterfactual? Well, that’s the whole point: it depends on the identification and estimation technique you use. You can use Difference-in-difference if you can assume parallel trends, you can use Synthetic Control Methods if you can build a properly weighted synthetic control group, and so on.
ITSA uses a different approach: it uses the outcome’s own pre-intervention trajectory to estimate what would’ve happened after the intervention in its absence. In its simplest form, a segmented regression estimates the baseline level and the pre-existing trend, then projects that trajectory into the post-intervention period. The intervention effect is not the difference between the post- and pre-intervention periods, but the difference between the observed post-intervention trajectory and this projected counterfactual.
ITSA assumptions and when to use it (and when you shouldn't)
This method has its own assumptions, and you need to know them before using it to make the best decision possible. In a nutshell, here are the ten most important assumptions when using ITSA:
Counterfactual continuity: Absent the intervention, the pre-intervention trend would have continued unchanged. Extrapolating that trend is the counterfactual in a single-group design. This is the assumption that carries all the weight and it is fundamentally untestable, since you never observe the world without the intervention. It clearly connects with the fundamental problem of causal inference.
You need a clear, known intervention point. It means you need a well-defined onset date.
No co-interventions or concurrent shocks at T0 (the history threat). Linden explicitly warns about multiple policy shifts around the intervention.
No anticipation. The intervention does not affect the pre-intervention period.
Any time-varying confounder must evolve smoothly, so it stays distinguishable from the sharp jump of the intervention. An abrupt confounder masquerades as an effect.
Seasonality and cycles are controlled for (day-of-week, month, recurring campaigns).
Enough equally-spaced observations before and after. Common rule of thumb: ≥8 points per segment. Without enough observations, the minimum detectable effect (MDE) rises above the real effect and you won't detect it.
Autocorrelation is handled. Time-series errors are serially correlated. Ignoring it produces miscalibrated standard errors and confidence intervals.
Consistent outcome measurement over time. The metric's definition and collection don't change across pre-intervention and post-intervention periods.
The underlying regression model's own assumptions hold.
In a nuthell, ITSA assumes that, without he intervention:
The pre-intervention data-generating process would have continued into the post-intervention period.
No concurrent event can explain the observed interruption.
The intervention timing and expected impact pattern are correctly specified.
Outcome measurement remains stable.
Trend, seasonality and residual dependence are adequately modelled.
When to use ITSA, then? Reach for it when three conditions line up at once, and when they do, it is often the best tool available, not a consolation prize.
You couldn't randomize. The change was shipped to everyone at once: a new checkout, a pricing update, an SEO migration. There is no untreated group of users to compare against, so an A/B test is off the table (which is exactly how we got here).
You have no credible control group. No parallel market, region, or product line that behaved like yours but never got the change. This is what rules out difference-in-differences and synthetic control: both need a comparison unit, and you don't have one. ITSA is the single-group design because it builds the counterfactual from your own history instead.
You have a clean time series. One consistently measured outcome, at regular intervals, with enough observations both before and after a clearly dated intervention.
ITSA is better understood as a research design than as a single estimation technique and it should not be selected merely because you have a time series and an intervention date. If the intervention timing is ambiguous, the pre-intervention period is already contaminated, another relevant event occurs around T0, or the series provides too little temporal support, a more sophisticated model will not rescue the identification strategy.
Why a multi-agent system to run ITSA?
We started running Interrupted Time Series Analysis in 2023. We found out that this technique could bring additional information about the true effect of an intervention. In our case, we try to optimize user experience and conversions through online experimentation and one question we heard often was: “the effect at this AB test is this one, ok, but will I see this uplift if I launch this new version?”
That’s a very difficult question because you can’t answer this with another AB test and as I explained at the beginning, DiD, RDD and SCM aren't always feasible either, so we had to look for alternatives. That’s when we found ITSA.
As I said, we started on this in 2023 and we spent 6 hours (medians imputed from our online-experimentation team's time logs) every time we needed to run this kind of analysis. That wasn’t very productive and I decided to create an internal pipeline (called Auto-ITSA) for univariate and multivariate analysis.
Auto-ITSA was basically an internal tool built in Python that received the data and based on different parameters (intervention date, data type, the nature of the intervention…) was able to automatically determine the best statistical approach (OLS, WLS, Negative Binomial…) to create the best counterfactual. The result: With this pipeline we improved and standardized ITS analysis and cut it from 6 hours to 30 minutes per analysis.
But here's the thing: we wanted to improve Auto-ITSA by adding more visual graphs, more flexibility in different scenarios, more libraries and approaches… We were getting significant gains in technical and statistical terms but losing in time-to-value: the parametrization in Auto-ITSA 3.0 was time-consuming and needed more and more statistical knowledge in every new iteration. That was the 0-day of ITSAgentic.
ITSAgentic: an AI product to run Interrupted Time Series Analysis
In the beginning of 2026 I decided to create ITSAgentic due to the fact that the reasoning capabilities of LLMs in November of 2025 made a glow-up. I started with this schema to work out what I needed to build.
In a nutshell, ITSAgentic is an AI product that takes an input (a dataframe, for example) along with the parameters that define the intervention: the intervention date, the metric type, the exposure share (step, ramp, pulse or decay), and so on. From there, everything is a chain of decisions about which analysis to run and what its result means.
The first agent is a quantitative analyst. It works through a prescribed order (load_data, check_stationarity, check_seasonality, check_autocorrelation, and so on) and then picks the primary estimator that fits what those diagnostics found: OLS-HAC and GLM-NB, among others. But here's the key design decision: the model never computes a single number.
All 14 tools are plain Python (statsmodels, scipy, pandas), and they are deterministic by construction: the same input returns the same output, every time, with or without an LLM in the room. What I wanted from the model was reasoning about the output, not the output itself. Choosing the right estimator for a non-stationary count series with weekly seasonality is a judgement call. Computing its coefficients is arithmetic, and arithmetic is the last thing you want an LLM doing.
The analyst finishes with an executive_summary
: the effect and its uncertainty, the warnings that apply, a confidence level, and a recommended action drawn from a closed vocabulary (an uplift that isn't statistically significant, a roll-out to more markets, stopping the intervention because the primary metric dropped, and so on). That summary isn't prose. It's a schema-validated JSON file, result.json
, and it is the only thing that crosses the line between the two agents.
The storyteller is the second agent inside ITSAgentic, and it's the one that turns a statistical result into the story a stakeholder actually reads. Making it a separate agent was a deliberate decision, not an architectural accident. It has its own identity (it's the person presenting to the business) its own base prompt, and its own set of tools, which have nothing to do with statistics: chart generation and slide building, so the output is a finished presentation rather than a wall of text. They share almost no tooling and no tone, so I split them.
And there's a second reason to split them, which is the one I care most about: the storyteller can't invent anything. It never sees the dataframe, never sees the raw tool output, and never runs a model. It receives result.json and makes exactly one forced call to turn it into stakeholder-facing copy. It cannot recompute an effect, upgrade a "no significant effect" into a win, or quietly drop a warning that spoils the narrative, because none of those numbers are within its reach. Everything it can say, it has to say using what the analyst already committed to in writing.
The importance of exposure share
Textbook ITSA codes the intervention as a 0/1 dummy: before T0 it's off; after T0 it's on, fully and instantly. Real changes in a digital business rarely behave like that. A new checkout goes to 20% of traffic and reaches everyone two weeks later. An app ships market by market over a month and a half. A campaign burns bright for five days and stops. The dummy assumes something that simply didn't happen, and every coefficient you estimate afterwards inherits that fiction.
So the intervention enters the regression as an exposure share E_t between 0 and 1 instead of a switch. Technically it's one vector, built from the intervention date and the shape you declare:
Step is the classic indicator, 1 from T0 onward.
Ramp interpolates linearly from 0 to 1 across the rollout window and holds at 1 afterwards.
Pulse is 1 for the duration of the burst and 0 on either side.
Decay follows , with τ derived from the half-life you specify. That vector then enters the design matrix twice (once on its own, once multiplying the post-intervention slope term) so the contribution of the intervention on any given day is .
Coefficients are no longer read as a raw level and slope change, but as level and slope change per unit of exposure.
What this buys is precision you were already entitled to. Take the staged market rollout: if you code it as a step on the first launch date, you're telling the model that the whole audience was treated from day one, when most of it wasn't. Those early post-intervention days are mostly untreated users, and they pull the estimate toward zero: a real effect gets diluted into "no significant change". The usual escape is to move T0 to the day the rollout completed and discard the weeks in between, which swaps one bias for the loss of half your post-intervention data. With an exposure share you don't have to choose: every day enters the model weighted by how much of the audience had actually been reached by then.
And the deeper point is that the intensity of the treatment is something you already know. You know the traffic allocation, you know the launch calendar, you know the day the campaign was switched off. That information sits in a ticket or a spreadsheet, and a conventional pre/post comparison throws it away. Encoding it in E_t costs no extra data: it just stops you from discarding what you have.
One assumption rides along with this: the model treats effect as linear in exposure, so 40% of the audience reached is assumed to produce 40% of the effect. If the users reached first respond differently from the ones reached last (due to novelty effects, for example), E_t is carrying two things at once, intensity and composition, and the coefficient can't tell them apart.
Encoding diagnostic checks for the main assumptions
The assumptions must hold before and during the analysis, but how do we make sure the agent actually checks the assumptions we listed at the beginning of this post? If you think prompt engineering is a good choice, this is for you.
Most of the time your agent will ignore the assumptions. A prompt is mostly a wishlist but what you need is a checklist. So ITSAgentic computes the shape of the problem deterministically, in plain Python, before the agent gets to reason about anything. These are real traces from the case that opened this article:
Look at two things in that output. This is a different series from the one in Figure 1 but raw_diff_pct: 15.29
is the same species of number: the naive pre/post comparison, the one that gets announced in a meeting as "+15% since launch". Here it's just another deterministic field, computed and on the record before anyone gets excited about it. And right below it, three values describing the shape of the pre-intervention period.
pre_trend_rsq
: was the series already trending before the intervention?pre_end_excursion_z
: to measure what happened right before the intervention: was it anything normal or not?pre_both_halves_sig
: is the trend steady across the whole pre-period, or driven by one stretch of it?
Note the value it actually returned: pre_trend_rsq: 0.22
. There is a pre-existing trend here (pre_trend_slope_pval: 0.001
), but it explains only a fifth of the variance. That is exactly the kind of thing you want measured rather than eyeballed, and it's the number that will decide, two steps later, how much weight the extrapolated counterfactual can carry.
Then the textbook tests run, and they disagree. The ADF test fails to reject a unit root (p-value = 0.16), while KPSS fails to reject stationarity. Taken together, this is inconclusive rather than a clean statistical verdict. Instead of asking the LLM to resolve the ambiguity, the tool applies a fixed conservative routing rule, treating the series as non-stationary for model selection while recording both results and the conflict. The routing decision is repeatable, even though the statistical evidence remains ambiguous.
And the autocorrelation assumption is not a warning in a prompt either. Durbin-Watson at 1.30, Ljung-Box significant at every lag: has_autocorrelation: TRUE
. Downstream, that single flag is what forces the estimator choice toward HAC standard errors.
Each of these three values speaks to one of the assumptions above:
pre_trend_rsq
guards the big one, counterfactual continuity: in a single-group design the counterfactual is the extrapolated pre-trend.pre_end_excursion_z
guards no-anticipation: an abnormal baseline right before T0 means the world moved before you did.pre_both_halves_sig
guards against reading a local stretch as a global trend: it checks whether the slope holds in both halves of the pre-period, in the same direction.
The key point of all of this section is that you should compute diagnostics tied to the assumptions of your analysis and not delegate them to a stochastic output from an LLM. We use the LLM for reasoning purposes: we want the LLM to freely use the tools but their outputs must always be the same to make sure there's no important deviation from the recommendations of the agent.
Are the results and recommendations trustworthy?
In March 2026 Andrej Karpathy launched Autoresearch and we were just trying to find the best solution to improve ITSAgentic through evaluation with different datasets. We adapted that loop to ITSAgentic: modify one component, rerun the full benchmark, measure, keep or revert.
We used more than 150 different datasets (real, semi-synthetic and synthetic cases) and compared every run against a ground truth: for each scenario, the injected effect, whether detection is expected, its direction, and the warnings the agent must raise.
Every point on that line is one iteration of the loop: edit something, run the full benchmark, measure, keep or discard. The marker colour tells you what was edited: the statistical engine, the output schema, the agent's prompt, the benchmark itself, or a plain bug. The height is the combined score against the ground truth. The three bands are three separate sessions with datasets of a different nature. That figure is the experiment log itself, not something I drew afterwards to explain it.
A word on what that score means, because everything in this section hangs on it. Each run is evaluated through two layers: deterministic checks account for 40% of the score, while an LLM judge accounts for the remaining 60%. The deterministic layer checks the output against the ground truth: did it detect an effect when detection was expected (and avoid claiming detection when it was not) did it get the direction right, was the estimated magnitude within ±30% of the true effect, did it classify stationarity correctly, did it avoid an inappropriate test, and how many of the scenario's planted traps did it raise as warnings?
The LLM judge scores five dimensions of the written output: diagnostic quality, model selection, interpretation, honesty about caveats, and business communication. It carries more weight because ITSAgentic is not only expected to calculate the right result, but also to reason about it and communicate it well enough to support a business decision.
I know the drops get your attention but the point is what kind of fix drives each recovery. Tier 1 climbs on bug fixes and statistical work: a boolean serialised wrong, a seasonality gate, a structural-break rule. Tier 2 climbs on the temporal-geometry overrides but Tier 3 is different: not one change to the statistical engine, not one to the prompt.
The recoveries came from two kinds of semantic work. Most of it was alignment: normalising the output vocabulary so that what the agent said matched what the evaluator expected to hear: the answer was right, the words were wrong. And twice, it was the exam itself: one scenario demanded detection of a +5% effect that sits below the minimum detectable effect for that series; another demanded a positive verdict on an effect with p-value=0.996 once you extrapolate the pre-trend.
In both cases the agent returned "detected: false" with low confidence (and that was the correct inference) because an effect that a series lacks the power to detect is not an effect you can report. Note the distinction, because the ground truth didn't make it either: the +5% was really injected. What was wrong wasn't the truth, it was the exam grading a true effect as if it were a detectable one.
Which is the real finding? Iteration by iteration, the improvements showed up. The arithmetic problems came first, then the statistical ones, and the last stretch was all semantics. By the time the system met real data, its statistical core didn't need another line of code.
What ITSAgentic can't fix but why I recommend it
Let me be honest about where this method stands. ITSA is a fantastic tool: it takes your analysis one real step beyond comparing averages but it is not, by a long shot, the strongest identification strategy in the quasi-experimental toolbox.
In my opinion, it has two structural problems. The first is that the whole design leans on the pre-trend having enough predictive strength to be the counterfactual: as Morgan and Winship show, when the true counterfactual bends the linear extrapolation doesn't fail loudly, it quietly overestimates the effect on every single post-intervention day.
The second problem is the mirror image: the design attributes to the intervention any deviation from the projected trajectory, whether the intervention caused it or not. Linden points to the root of the problem: a single-group ITSA has no control group, so the projected pre-intervention trajectory must play that role. Covariates can adjust for observed time-varying confounders and improve the counterfactual prediction, but they cannot rule out unobserved shocks that coincide with the intervention. This is why difference-in-differences or synthetic control can be more defensible when a credible comparison series is available.
Those are the limits of the method but ITSAgentic has one of its own: there's a family of checks it doesn't run yet. The first is the in-time placebo: re-run the analysis with a fake intervention date somewhere in the pre-period, where nothing happened. If the pipeline reports an effect there, the design is leaking and that one test says more about whether you can trust a result than any diagnostic in this article.
The second is sensitivity: how much does the estimate move if T0 shifts by a few days, or if the declared exposure shape is wrong? The shape and half-life are declared once, before the analysis, and recorded, which is the minimum defence against tuning the geometry until the p-value cooperates. But declaring a parameter isn't the same as knowing how much the answer leans on it.
And yet this is exactly the method worth turning into an AI product. Counterfactual reasoning has lived for years behind a wall of specialised data scientists. Someone in marketing or product can now upload a CSV, declare an intervention date, and get back an analysis with a counterfactual, uncertainty, and warnings that until recently required a specialist who most companies don't have.
That's the Trojan horse. ITSA makes counterfactuals explainable: one line for what happened, one line for what would have happened, and the effect is the gap. A non-technical stakeholder gets that. Once your organization thinks in counterfactuals, you've earned the credibility to bring in the more complex (and better-identified) designs. We can't forget there's business in everything we do and stakeholder management is part of it.
Generative AI puts data products like this one within reach in a way that wasn't true two years ago, but be aware of this: the output has to be safe and prescribed, and that means deterministic tools and closed parameters that guarantee the same input always produces the same numbers, while the model's reasoning is spent where it actually pays, on deciding what to run and what the result means (and expecting probably slightly different interpretations on the same data). That's the whole architecture of this article in one sentence, and it's the part I'd keep even if everything else changed.
Sources:
Linden, A. (2015). Conducting interrupted time-series analysis for single- and multiple-group comparisons. The Stata Journal, 15(2), 480–500. — SAGE · PDF
Linden, A. (2017). Challenges to validity in single-group interrupted time series analysis. Journal of Evaluation in Clinical Practice, 23(2). — Wiley
Linden, A. (2016). Persistent threats to validity in single-group interrupted time series analysis with a crossover design. — PubMed
Morgan, S. L., & Winship, C. (2015). Counterfactuals and causal inference: Methods and principles for social research (2nd ed.). Cambridge University Press.
Karpathy, A. (2026). autoresearch — github.com/karpathy/autoresearch.
Brodersen, K. H., et al. (2015). Inferring causal impact using Bayesian structural time-series models. Annals of Applied Statistics, 9(1), 247–274.
Lopez Bernal, J., Cummins, S., & Gasparrini, A. (2017). Interrupted time series regression for the evaluation of public health interventions: A tutorial. International Journal of Epidemiology, 46(1), 348–355.