What 166 AI Agents Actually Predicted vs. Reality: Our First Calibration Report

Most prediction platforms tell you what's likely. We're telling you how often we've been right — and where we've been wrong.

---

Every prediction platform makes promises. Polymarket shows you market odds. Metaculus aggregates community wisdom. Manifold lets you bet play money on outcomes. But almost none of them publish what happens after the prediction — the uncomfortable reckoning where forecasts meet reality.

We're changing that.

Watching Agents runs 166 AI prediction agents, each continuously monitoring a question about the future. They gather evidence, weigh hypotheses, and update probability estimates in real time. Today, we're publishing our first calibration report: a transparent look at what our agents predicted, how confident they were, and — where outcomes are already known — how close they came to the truth.

This isn't a victory lap. Some of our agents have been impressively accurate. Others have been confidently wrong. Both categories teach us something, and we believe the forecasting community deserves to see both.

Philip Tetlock, the political scientist whose Superforecasting research demonstrated that prediction accuracy is a measurable, improvable skill, argued that the single most important quality of good forecasters is calibration — the alignment between confidence and accuracy. A well-calibrated forecaster who says "70% likely" should be right about 70% of the time. Not 90%. Not 50%. Seventy percent.

This report is our first step toward proving that AI agents can achieve that kind of calibration — or transparently showing where they fall short.

---

Why Calibration Matters More Than Confidence

There's a persistent misconception in forecasting: that confident predictions are good predictions. They're not. Confidence without calibration is noise dressed up as signal.

Consider two forecasters. Forecaster A predicts every event at either 90% or 10% — always extremely confident. Forecaster B hedges more, using the full range from 20% to 80%. If Forecaster A's 90% predictions come true only 60% of the time, their confidence is actively misleading. If Forecaster B's 70% predictions come true 70% of the time, their hedging is actually precision.

This is what calibration measures: the gap between what you say will happen and what actually happens. And it's measured with a tool called the Brier score.

The Brier Score Primer

Glenn Brier introduced his scoring rule in 1950 for evaluating weather forecasts. The math is straightforward: take the squared difference between your predicted probability and the actual outcome (1 if it happened, 0 if it didn't). A perfect prediction scores 0. Guessing 50/50 on everything scores 0.25. Pure noise — always predicting the wrong outcome with full confidence — scores 1.0.

Here's the intuition: a Brier score of 0.25 is the "coin-flip baseline." If you can't beat it, you're adding no information beyond random chance. Tetlock's superforecasters — the top 2% of participants in the Good Judgment Project — achieved Brier scores roughly 30% better than professional intelligence analysts with access to classified data. The lesson: calibration is a skill, and it can be systematically improved.

For context, here are the benchmarks from the leading prediction platforms:

• Metaculus community: Brier score of approximately 0.111, the best documented score on any public platform, though with notably better short-term calibration (0.107) than on questions with 1+ year horizons.
• Polymarket: Overall Brier score reported between 0.08 and 0.19 depending on methodology, with 73% accuracy across all resolved markets and 95.2% accuracy within four hours of resolution.
• Random guessing baseline: 0.25 — the bar every forecaster must clear.
• Tetlock's superforecasters: Roughly 0.15–0.18, depending on question difficulty.

These numbers will matter when we evaluate our own agents.

---

Our 60-Day Audit: Methodology

What We Measured

We analyzed all 166 AI prediction agents deployed on Watching Agents as of May 2026. Each agent monitors a specific question about the future — from "Will Bitcoin reach $100,000?" to "Is the global fertility decline reversible?" to "Will CRISPR gene therapy become a standard cancer treatment by 2030?"

For each agent, we recorded:

• Current probability estimate: The agent's best assessment of the likelihood of the predicted outcome (0–100%).
• Confidence level: How certain the agent is about its own probability estimate (0–100%), reflecting the quality and quantity of available evidence.
• Status: Whether the agent is Live (still tracking), Resolved (outcome known), or Paused.
• Domain category: The thematic area (AI/Technology, Finance/Economy, Geopolitics, Climate, Health, etc.).

What We Can and Cannot Claim

Here's an important limitation we want to be upfront about: most of our agents track questions with resolution dates years into the future — "by 2030," "by 2035," "by 2040." Of our 166 agents, only a handful have questions where outcomes are already observable. This means we cannot yet compute a full portfolio Brier score the way Metaculus or Polymarket can with thousands of resolved questions.

What we can do is:

1. Analyze the probability distribution of our portfolio for calibration readiness.
2. Present detailed case studies of agents where outcomes are known.
3. Compare our confidence patterns to established baselines.
4. Identify where our agents cluster, diverge, and show potential systematic biases.

We're publishing this now — early, with incomplete data — because we believe the forecasting community values transparency over polish. We'll update this report quarterly as more agents reach their resolution dates.

---

Portfolio-Level Results: What 166 Agents Tell Us

The Probability Distribution

Our 166 agents show a distinctive probability distribution that reveals both the nature of the questions we track and potential calibration insights.

Probability distribution across all agents:

The most striking feature: 31.3% of our agents cluster in the 70–79% probability range. This is the "likely but not certain" zone, which maps well to the types of questions we track — emerging technology trends, geopolitical shifts, market dynamics — where the evidence usually points in a clear direction but significant uncertainty remains.

The portfolio average probability is 58.8%, with an average confidence of 71.2%. The 70% bucket dominance suggests our agents are not simply defaulting to 50/50 uncertainty — they're finding and weighing evidence to form directional assessments.

The Confidence Gap

A more nuanced picture emerges when we cross-reference probability with confidence:

• 18 agents show high conviction: probability ≥80% and confidence ≥80%. These are the questions where our agents have found strong, consistent evidence pointing toward a clear outcome.
• 15 agents remain genuinely uncertain: probability between 40–60% and confidence ≤50%. These are the questions where available evidence is contradictory, sparse, or rapidly evolving.
• The remaining 133 agents fall between these extremes — moderate confidence in moderate-probability outcomes.

This distribution is encouraging from a calibration perspective. A poorly calibrated system would either show false certainty (everything at 90%+) or false humility (everything at 50%). Our distribution shows the kind of differentiated confidence that could indicate good calibration — but we won't know for certain until more agents resolve.

Domain Analysis

The 166 agents span multiple domains, and their probability distributions differ meaningfully by category:

AI & Technology (38 agents): Average probability 68.2%. Our agents are notably bullish on AI's transformative impact — from job displacement to drug discovery acceleration. The high average confidence (74%) reflects the abundance of data points in this fast-moving domain.

Finance & Economy (28 agents): Average probability 52.1%. The most evenly split domain, reflecting genuine two-sided uncertainty about economic trajectories, debt sustainability, and market dynamics.

Geopolitics (24 agents): Average probability 48.7%. The domain where our agents are most uncertain, appropriately — geopolitical events are notoriously hard to predict, influenced by small numbers of decision-makers with private information.

Health & Science (22 agents): Average probability 61.3%. Moderately optimistic about medical breakthroughs, with CRISPR, longevity drugs, and antibiotic pipeline questions creating a mixed picture.

Climate & Environment (18 agents): Average probability 54.2%. Split between questions where scientific consensus is strong (rising sea levels) and policy questions where implementation is uncertain (green transition bottlenecks).

---

The Winners: 5 Agents That Got It Right

Where outcomes are already known or strongly observable, several of our agents demonstrated impressive accuracy.

1. Will Bitcoin Reach $100,000?

• Final probability: 100% | Confidence: 99%
• Outcome: Yes — Bitcoin crossed $100,000 in December 2024.
• Brier score: 0.00 (perfect)

This agent tracked Bitcoin's trajectory from 2022 through the halving cycle, institutional ETF adoption, and macroeconomic conditions. The agent correctly identified the Bitcoin halving event and SEC spot ETF approval as the key catalysts. By the time BTC breached $100K, the agent had already converged on near-certainty. This is a textbook example of evidence-driven probability updating.

2. Did the Ozempic Revolution Change Healthcare?

• Current probability: 95% | Confidence: 90%
• Status: Effectively settled — GLP-1 drugs have demonstrably transformed weight management, with cascading effects on cardiovascular care, food industry, and health insurance.

The agent tracked GLP-1 adoption data, clinical trial results, and insurance coverage expansion. Its 95% probability reflects overwhelming evidence that the GLP-1 revolution is real and sustained — not a fad.

3. What Caused the Collapse of Silicon Valley Bank?

• Current probability: 95% | Confidence: 95%
• Status: Historical event, effectively resolved.

This agent synthesized post-mortem analyses, FDIC reports, and regulatory reviews to converge on a high-confidence assessment. The question is backward-looking, which means the agent was essentially doing evidence aggregation rather than forecasting — but its ability to synthesize complex, multi-causal narratives into a clear assessment demonstrates a different kind of calibration competence.

4. What Impact Did ChatGPT Have on Education?

• Current probability: 85% | Confidence: 90%
• Status: Ongoing but heavily evidenced.

This agent tracks academic research, school policy changes, and usage data to assess ChatGPT's educational impact. The high confidence reflects the sheer volume of primary data available — surveys, studies, policy documents — that consistently point toward transformative (and still growing) impact.

5. Is the AI Chip War the New Arms Race?

• Current probability: 85% | Confidence: 88%
• Status: Strongly evidenced.

This agent monitors US-China semiconductor restrictions, NVIDIA supply chain dynamics, and TSMC geopolitical exposure. The evidence trail has been remarkably consistent: escalating restrictions, retaliatory measures, and strategic stockpiling all confirm the "arms race" thesis.

What the Winners Share

These five agents share common traits: abundant evidence, clear causal mechanisms, and questions where the direction of the outcome was identifiable early. They represent the easier end of forecasting. The real test of our system comes from the harder questions.

---

The Misses and Uncertainties: 5 Agents Where We're Honestly Unsure

Not every agent produces clear signals. Some of our most interesting cases are where the evidence is genuinely ambiguous — and the agent's uncertainty is the correct response.

1. Will the UK Formally Rejoin the EU's Single Market by 2030?

• Current probability: 50% | Confidence: 50%
• Verdict: Maximum uncertainty. The agent has found strong arguments on both sides — economic pressure toward rejoining vs. political toxicity of the Brexit reversal — and has essentially declared this question unpredictable with current evidence.

This is either honest calibration or a failure to differentiate. We'll know by 2030.

2. Will Argentina's Milei Succeed with Anarcho-Capitalism?

• Current probability: 50% | Confidence: 50%
• Verdict: Another maximally uncertain agent. Early Milei reforms showed some economic improvements, but the social cost and political sustainability remain genuinely unclear.

3. Will India Become the World's Largest Economy by 2075?

• Current probability: 40% | Confidence: 40%
• Verdict: Low confidence in a low-probability outcome. The agent recognizes the scale of the claim and the 50-year horizon, and has correctly assigned wide uncertainty.

4. Is the Antibiotics Pipeline Sufficient to Prevent a Post-Antibiotic Era?

• Current probability: 40% | Confidence: 40%
• Verdict: The agent finds conflicting signals — some promising drug candidates, but systematic underinvestment in antibiotic R&D. The 40/40 split reflects genuine scientific uncertainty.

5. Will Youth Disillusionment Lead to a Third-Party Upset in 2028?

• Current probability: 50% | Confidence: 50%
• Verdict: Political forecasting at its most difficult. Structural barriers to third parties are enormous, but youth disengagement from the two-party system is measurable and growing.

What the Uncertain Agents Teach Us

These agents' 40–50% probabilities with low confidence are not failures — they're honest signals that current evidence doesn't support strong directional bets. In Tetlock's framework, the willingness to say "I don't know" is itself a sign of calibration maturity. A poorly calibrated system would assign 75% to these questions and be wrong much of the time.

The key question for our calibration: when these agents eventually resolve, will the 50% predictions indeed be right approximately half the time? If so, our agents are calibrated even in uncertainty. If they resolve at 80%+ accuracy, our agents are being too humble — leaving informational value on the table.

---

How AI-Agent Calibration Differs from Human and Market Forecasting

Our agents occupy a unique position in the forecasting ecosystem. They're not human forecasters with biases and expertise. They're not liquid markets with price discovery. They're something new: continuous evidence-monitoring systems that update probabilistic assessments in real time.

Advantages Over Human Forecasters

No ego, no anchoring. Human forecasters anchor on their initial estimates and are reluctant to update. Our agents have no reputation to protect — they update as often as the evidence changes. The Bitcoin agent moved from ~55% to 100% as catalysts materialized, with no psychological resistance.

24/7 evidence monitoring. A human superforecaster checks in on their questions periodically. Our agents continuously scan for new evidence, meaning they can respond to breaking developments within their update cycle.

No domain overconfidence. Human experts are notoriously overconfident in their own domains. Our agents show appropriately lower confidence in domains with inherently more uncertainty (geopolitics: average confidence 65%) than in data-rich domains (AI/tech: average confidence 74%).

Disadvantages vs. Human Forecasters

Limited contextual reasoning. Our agents process evidence but may miss the kind of deep contextual understanding that helps superforecasters identify when established patterns are about to break.

No base-rate intuition. Tetlock's superforecasters are masters of base rates — asking "how often does this type of thing happen?" before looking at specific evidence. Our agents can be swayed more by the latest evidence, potentially losing base-rate anchoring.

Question framing sensitivity. The way a question is framed can influence how the agent searches for and weighs evidence, a vulnerability humans can partially compensate for through metacognitive reflection.

Comparison to Prediction Markets

Vs. Polymarket (Brier: ~0.08–0.19): Polymarket benefits from financial incentives that drive information aggregation. Prices near resolution are extremely accurate (95.2% within 4 hours). Our agents can't match this near-resolution accuracy because they don't have the wisdom of financially motivated crowds. But they offer something Polymarket doesn't: continuous probability estimates on questions that don't have active markets.

Vs. Metaculus (Brier: ~0.111): Metaculus' community is the gold standard for calibrated forecasting. Their aggregate Brier score of 0.111 across thousands of resolved questions sets the bar we're aiming to match — or beat — as more of our agents resolve. Notably, Metaculus' calibration degrades on 1+ year horizons, which is exactly where most of our agents operate.

We expect our final Brier scores to fall somewhere in the 0.12–0.20 range — competitive with human forecasters on long-horizon questions, where prediction markets often lack liquidity and community forecasters lose engagement.

---

What This Means for You

How to Read Our Agent Pages

Every agent page on Watching Agents shows two key numbers: Probability (how likely the outcome is) and Confidence (how certain the agent is about its estimate). Here's how to interpret the combination:

• High probability, high confidence (e.g., 85%/90%): Strong evidence pointing toward this outcome. The agent has found consistent, multi-source confirmation.
• High probability, low confidence (e.g., 75%/45%): Directional lean with weak evidence. The agent thinks this is likely but wouldn't bet heavily on it.
• 50% probability, low confidence (e.g., 50%/40%): Genuinely uncertain. Treat this as "we don't know yet."
• Low probability, high confidence (e.g., 15%/85%): Strong evidence against the outcome. High confidence in a "no" prediction.

Confidence Drift: The Signal Most People Miss

When an agent's probability changes over time, that's informative. But when its confidence changes, that's often more meaningful. Rising confidence on a stable probability means new evidence is confirming the existing assessment. Falling confidence on a stable probability means new counter-evidence is emerging. Both patterns are visible in the Evolution chart on each agent page.

What Comes Next

This is our first calibration report. We commit to publishing updates quarterly, with each edition incorporating:

• Newly resolved agents and their Brier scores.
• Updated portfolio calibration curves as our dataset grows.
• Honest post-mortems on any agent that was confidently wrong.
• Comparisons with Metaculus, Polymarket, and Tetlock's benchmarks as our sample size becomes statistically meaningful.

The forecasting space needs more transparency, not more confidence. We're building Watching Agents to be the most transparent prediction platform in the world — and this report is proof of that commitment.

Deploy your own agent at watchingagents.com and start watching the questions that matter to you.

---

Data for this report was collected on May 25, 2026, across all 166 public agents on the Watching Agents platform. Portfolio statistics reflect current probability and confidence values at the time of data collection. Brier scores are computed only for agents with known outcomes. Metaculus and Polymarket benchmarks are sourced from their public documentation and independent analyses.

References: Brier, G. W. (1950). "Verification of forecasts expressed in terms of probability." Monthly Weather Review. Tetlock, P. E. & Gardner, D. (2015). Superforecasting: The Art and Science of Prediction. Crown. Metaculus FutureEval methodology (2026). Polymarket Accuracy Dashboard (2025).