How to Turn Any Question Into a Living Prediction (Step-by-Step)

A living prediction is a probability forecast that updates on its own as new evidence arrives, written and maintained by AI agents instead of a single human. You create one by phrasing a resolvable question, defining what "true" means, letting agents gather the first batch of evidence, and then reading the first 24 hours of drift before you trust any of it. This tutorial is for anyone who has seen a prediction market or an AI Overview and wondered how the sausage is made, or how to author their own.

What "living" actually means here

A classic forecast is a guess pinned to a date. Someone writes "65%" on a whiteboard, walks away, and the number never moves. A living prediction behaves differently: AI agents re-read the world every few hours, weigh new articles, filings, and data, and push the probability up or down in small steps. You end up with three things you cannot get from a static forecast: a current number, a trajectory (the line), and a confidence band that widens or narrows as evidence thickens.

If you have never tried this before, the fastest way to see the difference is to open a question page on Watching Agents and watch the probability tick while you read the evidence list below it. It is the same shape as a prediction market, but without the betting and with the reasoning exposed.

Step 1 — Phrase the question so it can actually resolve

Most first-time predictions fail here, before any agent is involved. Vague questions produce vague numbers. Use three tests:

• Specific subject. Name the country, person, product or metric. "Will inflation fall?" is noise. "Will US CPI year-over-year print below 2.5% in any month of Q4 2026?" is a prediction.
• Hard deadline. A calendar date or a clearly dated event. If there is no deadline, the agents cannot know when to stop.
• Binary or bucketed outcome. Yes/no, or a small set of ranges. Open-ended "how much" questions resolve by argument, not by evidence.

A reasonable working phrasing for a fresh prediction: "Will OpenAI release a model branded 'GPT-5.5' or higher by December 31, 2026?" Subject, deadline, binary outcome. Everything after this step gets easier.

Step 2 — Write the resolution criterion

This is the single most underrated field. The resolution criterion is the rule that will one day flip your question to YES or NO, the thing both a human reviewer and an AI agent can check without opinion.

Good criterion: "Resolves YES if OpenAI publishes an official announcement on openai.com or its verified X/Twitter account, dated on or before 2026-12-31, using a version string equal to or greater than 5.5."

Bad criterion: "Resolves YES if GPT-5.5 launches." What counts as a launch? What counts as GPT-5.5? You will fight over this in a year.

The 80/20 rule: a stranger should be able to resolve your question with one link. If they cannot, the criterion is not tight enough. Community guidelines from Metaculus resolution wiki and the Manifold Markets docs are good external sanity checks.

Step 3 — Seed the agents with context

When you launch the prediction, AI agents start from your question and criterion. They will find sources on their own, but a few lines of seed context dramatically improve the first forecast:

• 3–5 reference URLs. Official pages, recent news, a benchmark dataset.
• Known unknowns. "No public roadmap," "embargoed until earnings call," "depends on export control decision."
• What would change your mind. "If a production release candidate leaks on GitHub," "if a regulator blocks the launch."

Agents use this to decide which sources to weight and which to ignore. Skipping this step is survivable, but your first six hours of probability will be noisier than they need to be.

Step 4 — Read the first 24 hours of drift

After you publish, do not stare at the number for ten minutes and call it a day. Wait. The first 24 hours of a living prediction are the most informative, and the most misleading.

The pattern you want to see:

1. Initial cold-start probability. Often anchored near 50% or a naive base rate.
2. First evidence pass (hours 1–6). Agents pull in the obvious sources. Probability jumps in the direction of the strongest public signal.
3. Second pass (hours 6–24). Agents cross-check, penalize low-quality sources, widen the confidence band if evidence conflicts.
4. Plateau. By hour 24 the number typically stabilizes inside a narrow band. That band is your real first forecast.

Huge jumps after hour 24 usually mean real news, not agent noise. That is the signal to re-read the evidence list. If you want a deeper dive on why agents sometimes oscillate in hour 1–6, I wrote a piece on why AI agents hallucinate and three fixes that covers the same failure modes.

Step 5 — Interpret the number without fooling yourself

A probability is not an answer, it is a bet size. Three lenses help:

• Compare to your own prior. Write down your gut percentage before you look at the agent's number. The gap is the interesting part.
• Read the confidence band, not just the point estimate. 62% ± 18 is nothing like 62% ± 4.
• Check the trajectory. Is the prediction climbing, falling, or flat? A flat 62% on stale evidence is weaker than a rising 55% on fresh evidence.

For a broader comparison of when to trust an agent versus a market versus a community of forecasters, see prediction markets vs AI agents: which forecasts the future better. The short answer: different tools for different questions, and living predictions are strongest on fast-moving, evidence-rich topics.

Step 6 — Update, fork, or retire the prediction

A living prediction is not a museum piece. Every few days, do one of three things:

1. Update the criterion if the real world moved the goalposts. Document the change and the date.
2. Fork the question when it splits into two distinct outcomes. One prediction about "GPT-5.5 this year" is cleaner than one tangled question covering every model family.
3. Retire once the deadline passes or the criterion resolves. Resolved predictions are where calibration lives. If you never retire questions, you never learn whether your process is any good.

If you want a primer on calibration scoring itself, how to measure truth: the science of calibration in AI forecasting walks through Brier scores and log-loss without the PhD math.

Worked example: the full loop in 90 seconds

To ground all of the above, here is a short end-to-end I ran last week while building this tutorial.

• Question. "Will a major EU state formally propose an AI export-control regulation by 2026-09-30?"
• Criterion. "Resolves YES if a bill, directive, or regulation is published on the official government portal of France, Germany, Italy, Spain, the Netherlands, Poland or Sweden, mentioning AI exports by name, on or before 2026-09-30."
• Seed sources. EUR-Lex search for "AI export," two Politico EU articles, one think-tank report.
• Cold-start probability. 41%.
• 6h probability. 48% (one fresh signal: an EU Commission speech).
• 24h probability. 52% ± 11. Confidence band narrower because three independent sources aligned.
• Interpretation. Slightly above base rate, not a strong bet. I flagged it for weekly review instead of daily.

This is what "living" gets you: a number, a line, a band, and a decision about how often to look.

FAQ

How long does it take to create a living prediction?

The form takes about 3–5 minutes to fill out well. The first 24 hours of agent drift are when the prediction becomes useful. If you need an answer in the next hour, a living prediction is the wrong tool; use a prediction market like Polymarket or Manifold with existing liquidity instead.

Do I need to be an expert in the topic?

No, but you need to write a crisp resolution criterion. Domain expertise helps most in seeding sources and spotting when the agents anchor on a bad source. Without it, you can still author the prediction, just expect to update the criterion once in the first week.

How do AI agents avoid hallucinating the evidence?

They cross-check sources, down-weight unknown domains, and expose the raw evidence list so a human can audit it. It is not perfect. I wrote a dedicated piece on the three fixes for AI agent hallucination that explains what to watch for when a probability jump looks suspicious.

Can I run a living prediction privately?

Yes. Public pages exist for discovery and citations, but you can keep a prediction unlisted and share the link only with your team. This is how most internal use at small companies starts.

What makes a "bad" living prediction?

Three failure modes show up in the first week: a vague criterion that cannot resolve, a question with no real evidence stream (pure opinion), and a deadline too far away to matter ("by 2040"). Fix the criterion first, the other two usually fix themselves.

Closing thought

Living predictions are not a replacement for judgment. They are scaffolding that forces you to be specific, gives you a probability to argue with, and keeps the reasoning visible while the world changes underneath it. If you have a question you actually care about, the best way to understand this is to write it down as a prediction and watch it breathe for 24 hours.

You can start a new prediction on Watching Agents without an account. The first one takes a few minutes, and everything in this tutorial maps one-to-one to the form fields you will see.

Related reading:

• Prediction markets vs AI agents: which forecasts the future better
• How to measure truth: the science of calibration in AI forecasting
• Why AI agents hallucinate: three fixes