How to hire an AI agent developer: what to ask and what good looks like

Not always, and a good developer will tell you so. If the steps are known in advance, a deterministic workflow with one or two model calls is cheaper, faster and easier to test than an autonomous agent. Agents earn their complexity when the path genuinely varies from case to case: triaging messy inbound requests, researching across several sources, or working through a task where the next step depends on what the last one found.

Be wary of anyone who proposes an agent before they have looked at your workflow. The first useful deliverable is often a map of which steps should be automated, which need a model, and which should stay with a person.

• How will we know it works?

  Look for an evaluation set: real examples with expected outcomes, scored automatically and re-run on every change to prompts, tools or models.

• What data and tools will it have access to?

  Each tool scoped to the minimum it needs. Read and write access are separate decisions, and irreversible actions need human approval.

• How will we see what it did?

  Every run traced: inputs, each model call, each tool call, outputs, latency and cost, searchable after the fact.

• What happens when it’s wrong or stuck?

  Fallbacks, retries, confidence thresholds and a handoff to a person, not just the happy path.

• How do you handle prompt injection and sensitive data?

  Content the agent reads can contain instructions. Expect a concrete answer, not “the model handles it.”

• Where will it run, and who controls the accounts?

  Your cloud, your model-provider keys and your repository, with separate environments for testing and production.

• What happens when the model changes, or we switch providers?

  Provider-specific code kept thin, and the eval set used to prove a new model before switching.

• What do we get at the end?

  Code, prompts, eval sets, dashboards and a runbook your team can operate, all in accounts you own.

On security, the OWASP Top 10 for LLM Applications (2025 edition) is a useful, vendor-neutral reference, and prompt injection sits at the top of its list. Anyone who has put agents into production should be able to walk you through how they mitigate it.

The difference rarely shows in a live demo. It shows the first time a real input surprises the system.

• Demo-only builds. Impressive on curated examples, with no eval set and no answer to “how often does it fail?”
• Accuracy claims without a test set. A percentage means nothing unless you can see what it was measured on.
• No tracing or logging. If you can’t see what the agent did, you can’t debug it or trust it.
• Unscoped access. An agent with admin credentials and no approvals is a liability, however well it demos.
• Everything in their accounts. Code, keys or infrastructure you don’t control means you can’t leave.
• Agents for everything. Proposing autonomy where a simple workflow would do adds cost and new ways to fail.
• Unpaid test projects as the selection process. They reward polishing a demo, the skill that matters least here.

De-risk it in stages, with a decision point after each one:

• Discovery. Map the workflow, the data and the systems involved. Decide what should be an agent, a plain workflow, or left alone, and agree how success will be measured.
• Pilot. Build one agent for one real workflow, with an eval set, tracing and scoped access from the start. Run it alongside your team before giving it real responsibility.
• Production. Harden the failure paths, add monitoring and alerts, then roll out to more volume or more workflows.
• Handoff or ongoing support. Your team takes over with documentation and a runbook, or the developer stays on part-time to keep improving it.

Fix the success criteria before the pilot starts. “It looks good” is not a criterion. “Handles these cases correctly, escalates those, at this cost per task” is.