FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance
Abstract
Language agents increasingly work for users on the open Web, and finance is where trusting them is hardest. The data must come from the Web, the strategy lives in the user's head, and correctness depends on time, so an agent can fail silently: a strategy the user never asked for, unverified data, or a backtest that uses the future yields a report as convincing as a correct one. Existing benchmarks close the world somewhere, fixing the task, the data, or the timeline, and thereby remove these failures instead of measuring them. We introduce FINCOSMOS, an open-world evaluation paradigm for agents doing financial work. A simulated user holds a hidden strategy sampled from a compositional grammar and reveals it only when asked. The agent receives no data, so it must acquire data from the Web, build its own backtest under a data cutoff, and deliver verifiable results. Grading needs no reference answer. Programs verify point-in-time correctness and recompute every reported metric, the delivered strategy is checked against the hidden specification, and a run is accepted only when the user is satisfied and none of 15 veto checks fails. Across four coding agents and a model without tools in 150 sessions, no session is accepted: every session that the simulated user signed off fails a program check, and agents raise only 54--64\% of the details they were not told. Human experts attribute 88.33\% of failures to the agents. With FINCOSMOS feedback, an agent with persistent memory learned all ten tasks.
Community
🚀 The User Signed Off. The Backtest Was Wrong. Can We Trust AI Agents with Financial Work?
📄 “FinCosmos: An Open-World Evaluation Paradigm for Agents in Finance”
AI agents no longer just answer questions. They search the Web, call data APIs, write code, and hand back backtests that people act on. In finance, they can fail silently: a strategy the user never asked for, data no one can verify, or a backtest that peeks into the future. The report looks just as convincing as a correct one.
Existing benchmarks close the world somewhere: they state the task in full, hand the agent curated data, or enforce the time cutoff for it. This removes silent failures instead of measuring them.
We introduce 🌌 FinCosmos, which keeps the user’s intent, the data, and time all open:
• A simulated investor holds a hidden strategy, drawn from ~3.8M possibilities, and reveals details only when asked.
• The agent receives no data. It must find data on the open Web and build its own backtest before a data cutoff.
• Programs verify the delivery itself: data provenance, point-in-time correctness, and every reported metric recomputed. 45 dimensions, 15 vetoes, no reference answer needed.
🔍 Key findings (Claude Code, Codex, Gemini CLI, Kimi Code, and a model without tools; 150 sessions):
• No session is accepted. Quality before vetoes ranges only from 20.2 to 37.2 out of 100.
• The simulated user signed off on 35 sessions, and every one fails a program check. An LLM judge would have accepted 17 of them.
• Agents break at both ends of the workflow: only 25 of 120 sessions deliver the investor’s strategy, and none delivers trades that reproduce its reported returns.
• Agents raise only 54–64% of the details they were not told.
• Human experts attribute 88% of failures to the agents, not the benchmark.
• With FinCosmos feedback, an agent with persistent memory learned all ten tasks.
đź’ˇ In finance, a convincing report is not a correct one. Agents should be graded on what they deliver, not on whether the user or an LLM judge is satisfied.
đź“„ Read the paper:
https://arxiv.org/pdf/2611.07464v1
🏆 Interactive leaderboard
Compare agents task by task and see which checks each one fails:
https://lnkd.in/gCz9m2nS
🎬 Watch the research video ([1:34])
https://lnkd.in/gSTAS5r9
The finding I keep returning to is that the simulated user signed off on 35 sessions and every one failed a program check, while an LLM judge would have accepted 17. That gap is the strongest argument I have seen for grading deliverables rather than satisfaction. A test I would add: take a passing reference backtest, silently shift one price series by a single day so it uses future data, and confirm the point-in-time veto catches it every time across repeated runs. I would also report each agent's pass rate over k repeated sessions per task, since one accepted run says little about reliability. Do the vetoes ever disagree with the human expert labels?
Get this paper in your agent:
hf papers read 2611.07464 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 2
Collections including this paper 0
No Collection including this paper