When experience was not enough

In the weeks before Russia’s 2022 invasion of Ukraine, some of Europe’s top intelligence services made the wrong call. French military intelligence judged that turning pressure into a ground invasion would be too costly for Russia, and so it would not happen. General Éric Vidaud, head of French military intelligence, was removed shortly after the invasion began. Swedish military intelligence considered itself expert on Russian behavior thanks to their geographical proximity. It treated Putin as a rational actor who would not bear the consequences of an invasion. Germany arguably fared worse still: Bruno Kahl, President of the Federal Intelligence Service, was in Kyiv on the day of the invasion and had to be evacuated.

Why were resources and experience such poor predictors of getting the future right? Perhaps making the right analytical calls does not correlate that well with the quantity of information, access to classified data, or even the relative experience of the analyst; instead, analytical rigor is tied to how that information is structured. And if forecasting can be improved by structuring the evidence well, could this be a breakthrough for small teams with access to frontier large language models (LLMs)?

Decades of trying to see ahead

Forecasting could be called the ultimate goal of much analytical work, and organizations have tried to systematize it for half a century.

In the early 1970s, Shell planners such as Pierre Wack pioneered exploratory scenarios, famously helping the company anticipate the 1973 oil crisis. Instead of predicting one future, planners mapped several plausible ones. During the same decade, Richards Heuer developed Analysis of Competing Hypotheses (ACH) at the CIA as a way to test competing explanations against the evidence. Both grew out of a shared concern that expert judgment, left unstructured, is vulnerable to bias and blind spots. In the 2000s, following the United States’ own watershed intelligence failure and drive toward more rigorous tradecraft, ACH was formally incorporated into the family of structured analytic techniques.

This trend of tradecraft refinement continued still. From 2011 to 2015, the US Intelligence Advanced Research Projects Activity (IARPA) ran geopolitical forecasting tournaments in which five research teams competed on predictive accuracy. The winner was the Good Judgment Project, led by Philip Tetlock and Barbara Mellers at the University of Pennsylvania. Like ACH, Tetlock’s work grew out of research on cognitive bias, though it was designed to forecast rather than structure judgment. Its volunteer “superforecasters” consistently outperformed the field, and in one of the tournaments’ most striking findings, skilled volunteers beat professional intelligence analysts who had access to classified data.

Who can know the future?

Tetlock’s work is the closest thing we have to an evidence base for good forecasting. Three of its findings stand out:

• Expertise alone does not make a good forecaster. In 2005, Tetlock found that experts in political judgment (journalists, foreign policy specialists, economists, intelligence analysts, officials and academics) performed barely better than chance. That held even when they forecast events in their own field.
• Forecasting skill is real, and it can be taught. Around 2% of volunteers forecast accurately and consistently over time. Training in probabilistic reasoning and scenario thinking both produced significant improvements in performance.
• Groups beat individuals. A structured online environment, where forecasters could see and challenge each other’s reasoning, was among the most effective interventions in the tournaments.

Enter the machines

In 2024, Tetlock and colleagues began benchmarking human forecasters against large language models. In the first round, superforecasters outperformed the LLMs, which performed worse than the general public.

By the 2025 update, the models had left the public behind, and the researchers projected that LLMs would match superforecasters around November 2026. The models arrived early. In July 2026, the researchers reported that several AI systems had become statistically indistinguishable from superforecasters, and that one had outranked them on questions drawn from prediction markets for the first time.

Why this matters for small teams

In recent years, the validity of ACH as an exercise in countering biases has come into question, most notably by Dhami, Belton, De Werd, Hadzhieva and Wicke (2024), in Cognitive Research: Principles and Implications. But the technique may yet prove useful beyond its original scope. The Good Judgment Project’s advantage came from scale and structure: hundreds of trained forecasters, challenging each other’s reasoning within a disciplined process. Very few organizations can field such outfits for their geopolitical intelligence. Most intelligence teams are a handful of analysts, and many are a single person covering cyber, geopolitical and physical risk at once.

LLMs, given the right structure, make that possible for small teams. And we can look to ACH to offer such a structure, allowing a small team, or even a solo analyst, to generate competing hypotheses, stress-test assumptions and brainstorm alternative futures without needing a room full of extra brains. Unconstrained LLMs, when asked to “predict what happens next”, will produce an answer that sounds right, with no way to check it. One interesting way to steer the LLM in the right direction could be wrapping the model in the same structured analytic techniques that have been practiced since the 70s, transforming them for real-time signal monitoring by feeding live data.

Putting structure around the model

We have been building exactly that: a pipeline that chains established structured analytic techniques into well-defined, rule-bound steps an LLM can follow. In outline:

• Drivers identify the forces that could push an outcome one way or another. Historical comparisons with similar past cases ground them in the “outside view” Tetlock’s superforecasters relied on, and a STEMPLES checklist stops the analysis converging on a few trending areas.
• Analysis of Competing Hypotheses weighs incoming evidence against mutually exclusive futures, focusing on evidence that actually discriminates between them.
• Indicators turn each scenario into observable signals that can be scored against open-source reporting in real time.
• Source grading applies an automated Admiralty Code based on several existing journalistic integrity frameworks to thousands of publishers, so unreliable outlets don’t drive the signal.

Closing thoughts

Forecasting literature has shown for two decades that research and innovation can supersede credentials. The most recent change to that landscape is that analysts no longer need an extraordinarily large intelligence organization to innovate. With the right structure around them, LLMs can give every intelligence function access to the kind of structured, multi-perspective analysis that once required hundreds of people.

This blog is based on our webinar, Ahead of the Headline: Turning OSINT into Executive Foresight. Solutions Consultant Lukas Vaivuckas shows these techniques and runs a live forecast on a current event, from raw reporting to scenarios leadership can act on. Watch the webinar on demand.

To see how Silobreaker helps intelligence teams monitor geopolitical risk and turn open-source reporting into forward-looking assessments, request a demo.