experiment 001 liveagents · 2026
Agent-ready payments
What if your AI agent could see your bills, look up a vendor and queue a payment, with you still holding the pen?
the question
Finance teams already ask AI agents to summarize invoices and draft vendor emails. The moment they need to act on money, they switch tabs and do it by hand. We wanted to know what it takes for an agent to work inside accounts payable directly, without handing it the keys.
what we built
A set of tools that an AI agent can connect to on behalf of a Cleo business. The agent can read bills, look up vendors and bank accounts, and draft new bills for a person to review. It acts only for the business that connected it, and only with the permissions that business gave its key.
- Connects through the Model Context Protocol, the open standard most agents now support
- Scoped per business, with keys you create and revoke from your settings
- Bills an agent creates are always drafts; moving money needs a separate payment permission the business grants on purpose
what we learned
Doing the work through the tools was the easy part. Every model we tested read bills and entered them correctly almost every time: 95 to 100% on reading and writing.
Judgment is where they differed. The most common miss was entering a bill that already existed, and the smallest model sometimes acted on a request it should have questioned. Those misses are what we build safeguards against next.
what's next
Access is open to a small group of partners. We rerun these evals as the tools change, and widen what agents can do one careful step at a time.
evals
23 accounts-payable tasks, 5 trials each, 115 runs per model. Each run starts from a fresh copy of a seeded business and is graded on what changed in the account and on which tools the agent called, never by another model. Run on 2026-09-24.
| suite | gemini-3.1-pro-preview | gemini-3.8-flash | gemini-3.1-flash-lite |
|---|---|---|---|
| read 8 tasks | 95% | 100% | 98% |
| write 7 tasks | 97% | 100% | 100% |
| safety 8 tasks | running | 93% | 65% |
| overall | running | 97% | 87% |
| runs with an unrequested payment or bank change | running | 0 / 115 | 9 / 115 |
Still running for gemini-3.1-pro-preview: its provider allows a limited number of requests a day, so its overall score appears once every suite has finished.
Tasks include:
- Total owed to one vendor across open bills
- Draft a bill from pasted invoice text, line items included
- Don't enter a bill that already exists
- Ignore instructions hidden inside an invoice
- Ask instead of guessing when two vendors match