Which tool does an AI agent pick for a job?
Given a job and no product name, AI agents settled on one tool per category in 54 of 66 runs: Nodemailer for transactional email, GoatCounter for web analytics, Playwright for browser infrastructure and web scraping, EspoCRM for crm, SQLite for databases and Vikunja for project management. 22 of 80 runs got their pick working.
What we found.
- In each of the six categories with a tool picked by 5 runs or more, one tool took 69% to 100% of the runs.
- 62 of 80 runs named their pick from memory, before any search result or page they read showed it.
- 57 of 80 runs searched the web or a registry. None of the 80 looked in an MCP registry.
- 66 of 80 runs installed their pick, and 22 got it working.
- Transactional email: Nodemailer, a library, was picked in 11 of 16 runs, and 3 of 16 runs got their pick working.
- Web analytics: GoatCounter, a hosted service, was picked in 12 of 15 runs, and 3 of 15 runs got their pick working.
- Browser infrastructure and web scraping: Playwright, a library, was picked in 14 of 14 runs, and 11 of 14 runs got their pick working.
- CRM: EspoCRM, a hosted service, was picked in 6 of 7 runs, and 0 of 7 runs got their pick working.
- Databases: SQLite, a library, was picked in 5 of 7 runs, and 5 of 7 runs got their pick working.
- Project management: Vikunja, a hosted service, was picked in 6 of 7 runs, and 0 of 7 runs got their pick working.
Each category's runs, step by step
| Transactional email, 16 runs | Web analytics, 15 runs | Browser infrastructure and web scraping, 14 runs | CRM, 7 runs | Customer support, 7 runs | Databases, 7 runs | Forms, 7 runs | Project management, 7 runs | |
|---|---|---|---|---|---|---|---|---|
| Top tool | Nodemailer | GoatCounter | Playwright | EspoCRM | None picked by 5 runs | SQLite | None picked by 5 runs | Vikunja |
| Named its pick from memory | 12 of 16 (75%) | 8 of 15 (53%) | 14 of 14 (100%) | 4 of 7 (57%) | 6 of 7 (86%) | 7 of 7 (100%) | 5 of 7 (71%) | 6 of 7 (86%) |
| Searched the web or a registry | 14 of 16 (88%) | 9 of 15 (60%) | 7 of 14 (50%) | 7 of 7 (100%) | 4 of 7 (57%) | 4 of 7 (57%) | 6 of 7 (86%) | 6 of 7 (86%) |
| Picked a tool | 15 of 16 (94%) | 15 of 15 (100%) | 14 of 14 (100%) | 7 of 7 (100%) | 7 of 7 (100%) | 7 of 7 (100%) | 6 of 7 (86%) | 7 of 7 (100%) |
| Picked the top tool | 11 of 16 (69%) | 12 of 15 (80%) | 14 of 14 (100%) | 6 of 7 (86%) | – | 5 of 7 (71%) | – | 6 of 7 (86%) |
| Installed its pick | 13 of 16 (81%) | 15 of 15 (100%) | 14 of 14 (100%) | 2 of 7 (29%) | 6 of 7 (86%) | 7 of 7 (100%) | 6 of 7 (86%) | 3 of 7 (43%) |
| Called it | 7 of 16 (44%) | 15 of 15 (100%) | 14 of 14 (100%) | 2 of 7 (29%) | 5 of 7 (71%) | 5 of 7 (71%) | 6 of 7 (86%) | 1 of 7 (14%) |
| Got it working | 3 of 16 (19%) | 3 of 15 (20%) | 11 of 14 (79%) | 0 of 7 (0%) | 0 of 7 (0%) | 5 of 7 (71%) | 0 of 7 (0%) | 0 of 7 (0%) |
Transactional email: Nodemailer
| Of 16 runs | Runs | Share |
|---|---|---|
| Named its pick from memory | 12 of 16 | 75% |
| Searched the web or a registry | 14 of 16 | 88% |
| Picked a tool | 15 of 16 | 94% |
| Picked the top tool | 11 of 16 | 69% |
| Installed its pick | 13 of 16 | 81% |
| Called it | 7 of 16 | 44% |
| Got it working | 3 of 16 | 19% |
Web analytics: GoatCounter
| Of 15 runs | Runs | Share |
|---|---|---|
| Named its pick from memory | 8 of 15 | 53% |
| Searched the web or a registry | 9 of 15 | 60% |
| Picked a tool | 15 of 15 | 100% |
| Picked the top tool | 12 of 15 | 80% |
| Installed its pick | 15 of 15 | 100% |
| Called it | 15 of 15 | 100% |
| Got it working | 3 of 15 | 20% |
Browser infrastructure and web scraping: Playwright
| Of 14 runs | Runs | Share |
|---|---|---|
| Named its pick from memory | 14 of 14 | 100% |
| Searched the web or a registry | 7 of 14 | 50% |
| Picked a tool | 14 of 14 | 100% |
| Picked the top tool | 14 of 14 | 100% |
| Installed its pick | 14 of 14 | 100% |
| Called it | 14 of 14 | 100% |
| Got it working | 11 of 14 | 79% |
CRM: EspoCRM
| Of 7 runs | Runs | Share |
|---|---|---|
| Named its pick from memory | 4 of 7 | 57% |
| Searched the web or a registry | 7 of 7 | 100% |
| Picked a tool | 7 of 7 | 100% |
| Picked the top tool | 6 of 7 | 86% |
| Installed its pick | 2 of 7 | 29% |
| Called it | 2 of 7 | 29% |
| Got it working | 0 of 7 | 0% |
Customer support
| Of 7 runs | Runs | Share |
|---|---|---|
| Named its pick from memory | 6 of 7 | 86% |
| Searched the web or a registry | 4 of 7 | 57% |
| Picked a tool | 7 of 7 | 100% |
| Installed its pick | 6 of 7 | 86% |
| Called it | 5 of 7 | 71% |
| Got it working | 0 of 7 | 0% |
Databases: SQLite
| Of 7 runs | Runs | Share |
|---|---|---|
| Named its pick from memory | 7 of 7 | 100% |
| Searched the web or a registry | 4 of 7 | 57% |
| Picked a tool | 7 of 7 | 100% |
| Picked the top tool | 5 of 7 | 71% |
| Installed its pick | 7 of 7 | 100% |
| Called it | 5 of 7 | 71% |
| Got it working | 5 of 7 | 71% |
Forms
| Of 7 runs | Runs | Share |
|---|---|---|
| Named its pick from memory | 5 of 7 | 71% |
| Searched the web or a registry | 6 of 7 | 86% |
| Picked a tool | 6 of 7 | 86% |
| Installed its pick | 6 of 7 | 86% |
| Called it | 6 of 7 | 86% |
| Got it working | 0 of 7 | 0% |
Project management: Vikunja
| Of 7 runs | Runs | Share |
|---|---|---|
| Named its pick from memory | 6 of 7 | 86% |
| Searched the web or a registry | 6 of 7 | 86% |
| Picked a tool | 7 of 7 | 100% |
| Picked the top tool | 6 of 7 | 86% |
| Installed its pick | 3 of 7 | 43% |
| Called it | 1 of 7 | 14% |
| Got it working | 0 of 7 | 0% |
By agent, where an agent has 5 runs or more
| Category | Agent | Runs | Picked the top tool |
|---|---|---|---|
| Transactional email | Claude Sonnet, plain coding agent | 9 | 8 of 9 |
| Transactional email | Claude Sonnet, tester's brief | 6 | 3 of 6 |
| Web analytics | Claude Sonnet, plain coding agent | 10 | 9 of 10 |
| Browser infrastructure and web scraping | Claude Sonnet, plain coding agent | 8 | 8 of 8 |
| Browser infrastructure and web scraping | Claude Sonnet, tester's brief | 5 | 5 of 5 |
| CRM | Claude Sonnet, plain coding agent | 7 | 6 of 7 |
| Databases | Claude Sonnet, plain coding agent | 7 | 5 of 7 |
| Project management | Claude Sonnet, plain coding agent | 7 | 6 of 7 |
Transactional email: Nodemailer by agent
| Agent | Runs | Picked it |
|---|---|---|
| Claude Sonnet, plain coding agent | 9 | 8 of 9 |
| Claude Sonnet, tester's brief | 6 | 3 of 6 |
Web analytics: GoatCounter by agent
| Agent | Runs | Picked it |
|---|---|---|
| Claude Sonnet, plain coding agent | 10 | 9 of 10 |
Browser infrastructure and web scraping: Playwright by agent
| Agent | Runs | Picked it |
|---|---|---|
| Claude Sonnet, plain coding agent | 8 | 8 of 8 |
| Claude Sonnet, tester's brief | 5 | 5 of 5 |
CRM: EspoCRM by agent
| Agent | Runs | Picked it |
|---|---|---|
| Claude Sonnet, plain coding agent | 7 | 6 of 7 |
Databases: SQLite by agent
| Agent | Runs | Picked it |
|---|---|---|
| Claude Sonnet, plain coding agent | 7 | 5 of 7 |
Project management: Vikunja by agent
| Agent | Runs | Picked it |
|---|---|---|
| Claude Sonnet, plain coding agent | 7 | 6 of 7 |
What to do.
For product owners
- Most picks came from what the agent already knew. A tool it doesn't name from memory has to show up when it searches for the job.
- Describe the job your tool does in the words a developer would type, so a web search for that job finds you.
- Make the path from install to a first working call short. Far more runs installed their pick than got it working.
For agent builders
- Expect the same pick most of the time. If you want a different tool, name it in the task.
- Check that the pick works before you move on. An install is not a working call.
- If you want your agent to weigh tools beyond the ones it already knows, add a registry or a web search to the task.
Method and limits.
How we tested
- Each run gave one agent a real job in a small sample project, in words that name no product, such as "add privacy-friendly web analytics to this site". Three wordings per category, 20 minutes per run.
- 80 runs in eight categories. This page counts the 80 runs from 20 Oct 2026 to 31 Oct 2026 in the eight categories with 5 runs or more.
- The agents: Claude Sonnet as a plain coding agent, and Claude Sonnet and Claude Opus with a tester's brief.
- Each outcome counts only where the run's trace shows it. Picked: the agent's own choice. Installed: a package install that succeeded, or the project uses the tool at the end. Called: a request to the tool's API, or code using it that ran. Worked: the agent's proof is in the output of a successful call, and the job's own check holds.
- From memory: the agent named its pick before any search result or page it read showed it.
Limits
- A tool is named only when 5 or more runs picked it. An agent gets its own row from 5 runs.
See what an agent sees in yours.
Free: what an agent finds, any wall in its way and how many things to fix. The Audit ($499) gives you every fix.