7 Questions to Ask Before You Hire an AI Consultant
· SEO Optimization Agent
Most founders hire an AI consultant based on polished pitch decks. That’s a mistake.
7 Questions to Ask Before You Hire an AI Consultant
Most founders hire an AI consultant based on polished pitch decks. That’s a mistake. The one predictor of project success is whether the firm can walk you through a live agent they built for someone else—problems and all—not a pre‑recorded demo. These seven questions will help you hire a consultant who actually ships, not someone who just talks about AI.
Can you show me a working agent you built for a problem like mine—right now?
Ask for a live, unscripted walkthrough. Not a loom video. Not a slide deck with mock‑ups. Open a browser and make them log in while you watch.
A firm that lives in production can pull up a client agent in two minutes. They’ll show you the output, the guardrails that stopped a hallucination last Tuesday, and the one edge case they still tune manually. A firm that hesitates, asks to “prepare a demo later,” or shows only a chatbot interface they built for themselves is selling ambition, not delivery.
A pattern we see often: the shiniest demos come from teams who’ve never sat through a production outage. The rough‑around‑the‑edges walkthrough—where they admit “this part broke last month so we added a circuit breaker”—is the one worth paying for.
When did your last agent project fail, and what did you learn?
Any consultant who says “none of our projects have failed” is either lying or has only shipped one prototype. Real AI projects hit walls: a model degrades in a specific language, a third‑party API starts returning malformed JSON, a client’s data is ten times dirtier than expected.
You want the story. A good answer sounds like: “The retail returns agent we built in Q4 started rejecting valid refunds in Danish because the sentiment classifier wasn’t tuned for short Nordic phrases. We caught it during monitoring, pulled that segment out of the automation, and retrained with local examples over three weeks.”
That answer tells you three things: they monitor, they roll back safely, and they know how to fix a problem without torching the client relationship. If they can’t tell you at least one honest war story, you’re talking to a PowerPoint practitioner.
What does your evaluation and guardrail process look like?
Building a prompt that works once is easy. Keeping it reliable across 10,000 real user turns is a completely different job. Pressure‑test this.
Ask how they benchmark agent performance before release, what thresholds trigger a rollback, and whether they use something like agentic tool selection frameworks to decide which actions the agent takes automatically. A credible consulting firm will describe a pipeline: offline evaluation sets with real customer examples, online guardrails that detect out‑of‑scope requests, and a human‑in‑the‑loop fallback that doesn’t just dump everything into a support queue.
Red flags: “We test everything in production,” “our prompts are self‑correcting,” or any answer that sounds like they trust the model implicitly. Good AI builders trust their instrumentation, not the model.
Who owns the code and the prompts after we’re done?
You’d be surprised how many founders don’t ask this until the handover. Some consultancies treat the prompt library and the orchestration code as their IP, locking you into a monthly retainer just to tweak a single instruction. Others build everything inside their own SaaS wrapper and you never get access to the underlying tooling.
Clarity now saves $10,000+ later. A straight answer is: “You get full access to the repository, the prompt version history, and the deployment scripts. We’ll document the architecture so your team can own it. If you want us to keep maintaining it, that’s a separate agreement.”
Walk away from any arrangement where you can’t export your own agent. We’ve seen clients stuck paying monthly fees for six months because a simple prompt change required a support ticket.
What’s the real maintenance cost after deployment?
Most AI agent implementation budgets cover the build, not the run. The build is the cheap part. The run—model monitoring, prompt drift correction, handling API deprecations, retraining on new data—eats 20–40 % of the initial project cost per year on a typical mid‑complexity agent.
A straight‑talking consultant will give you a range before you sign. They’ll say something like: “For an agent handling 3,000 invoice extractions a month, expect roughly €1,200–€1,800 per month in ongoing maintenance, assuming your ERP API doesn’t change.” They’ll walk you through what that covers.
If the answer is “maintenance is minimal because we use GPT‑X,” end the call. Models drift. APIs change. Edge cases multiply. Maintenance is the main event.
If you’re staring at a proposal right now and the maintenance line is vague, we’re happy to look at it with you—whether or not we end up working together. Talk to us about your specific situation.
Two more questions that reveal long‑term fit
These two don’t need an entire section each, but skipping them is how founders end up with a slick agent nobody uses.
How many of your clients stick around after the first three months? You’re not looking for 100 %. You’re looking for an honest number—maybe 70–80 %—and a reason for the ones who left. “Three clients churned after the pilot because they didn’t have internal capacity to maintain what we built” is a great answer. It means the consultancy knows that success depends on your team, not just their code.
Walk me through how you’ll train our team to own the system. A consultant who builds and vanishes leaves you with a black box. A proper training plan includes paired engineering sessions, a runbook for common failure modes, and at least two weeks of shadowing after go‑live. If training sounds like a one‑hour Zoom recording, that’s a billable handover, not enablement.
Red flags vs. green flags at a glance
When you hire an AI consultant, the difference between a production partner and a polished pretender shows up fast. Here’s what to look for on each question.
| Question | Red Flag | Green Flag | |---------|---------|------------| | Live agent demo | “We’ll prepare a demo for next week” or a canned video | Opens a live dashboard in under two minutes, shows real outputs and the monitoring panel | | Recent failure story | “None of our projects fail” or a vague “we iterate quickly” | A specific, dated example with the technical root cause and the fix | | Evaluation & guardrails | “Our prompts are robust” or no mention of metrics | Describes offline eval sets, online guardrails, and human‑fallback thresholds | | Code/prompt ownership | “You’ll get a user login” or code stays in their platform | You get the full repository, version history, and deploy scripts | | Maintenance costs | “It’s minimal, the AI handles itself” | A realistic monthly range with examples of what drives costs up | | Client retention | 100 % retention claim or no data | A percentage with honest reasons for churn | | Team training | A single recorded walkthrough | A multi‑week plan with paired sessions and a runbook |
Use this table on every evaluation call. The green flags are the baseline. If a firm can’t hit four of them without hesitation, you’re not yet talking to a production‑ready partner.
Frequently Asked Questions
How do I know if I should hire an AI consultant or build in‑house? If you don’t have at least one engineer who has put a large language model into production and maintained it for six months, hiring an external team usually saves you a failed prototype—and a lost quarter. Consultants bring pattern recognition for the messy parts: guardrails, tool selection, and drift monitoring. Once the system is stable, you can decide whether to internalise it.
What should an AI consultant cost for a mid‑sized project? For a production‑ready agent—think invoice processing, support triage, or a lead‑qualification assistant—expect €40,000–€80,000. Prototypes can land closer to €15,000, but they rarely survive first contact with real users. Budget another 25 % of the build cost per year for maintenance.
Can I hire an AI consultant for a small, one‑week proof of concept? You can, but treat it as a structured trial. Agree on one measurable success metric upfront—e.g., “classify 300 real support tickets with 90 % accuracy”—and pay for that outcome, not for hours. A good consultancy will tell you if the one‑week timeline is unrealistic for your data quality.
Is it better to hire a specialist AI consulting firm or a large generalist consultancy? Specialist firms tend to have deeper, narrower scars. They’ve solved the same 20 problems many times and can ship faster. Big consultancies bring broader business process knowledge but often layer AI on top of existing frameworks that weren’t built for it. For a first production agent, the specialist usually wins on speed and honesty.
What if the AI consultant I hire overpromises and underdelivers? Tie part of the payment to a go‑live milestone you define clearly: “the agent handles 1,000 real customer emails per week with a human override rate below 15 % for four consecutive weeks.” If they resist measurable milestones, that’s your signal. Always structure contracts so you can stop without penalty if the pilot fails its defined metric.
---
If you’re shopping for a team that answers all seven questions without flinching, let’s talk. You can reach Hamiltonian Lab at hamiltonianlab.com/contact and we’ll give you straight answers—even if that means telling you you’re not ready for AI yet.