How to Choose an AI Development Agency: 12 Questions That Separate Engineers from Resellers

Choosing an AI development agency is choosing who holds your data, your timeline, and your budget. In 2026 the market is flooded with "AI agencies" that are three people with ChatGPT Pro and a website. These 12 questions — grouped into four buckets — will separate the engineers from the resellers before you sign anything.

Capability: can they actually build?

1. What have you built in-house, not just for clients? Agencies that ship their own products understand engineering reality — maintenance, failure modes, real users. The best answer is a portfolio you can open and use.

2. Show me a production system, not a demo. Demos are rehearsed. Production systems have real users, real edge cases, and real scars. Ask for something live.

3. What's your evaluation story? How do they know the model is good enough? If the answer is "we tested it and it felt right," they don't have an evaluation story. Look for benchmarks, test sets, and a measurement culture.

Process: will you know what's happening?

4. What's the fixed price for my scope? Any agency that can't name a number after a real conversation either hasn't understood the work or plans to bill you by surprise. Fixed price, named before commitment, is the standard we hold ourselves to.

5. How will I see progress? If the answer is "weekly email updates," you're going to disappear into a black box. Look for a visible plan, visible progress, and a way to talk to the project without formal meetings.

6. Who's accountable when the model is wrong? A person. Not a prompt, not a platform, not a policy. The answer should name who owns quality.

Handling: what happens when it gets hard?

7. What's your fallback when the AI fails? Production AI fails. The question is whether the system degrades gracefully — fallbacks, human handoff, guardrails — or falls over loudly.

8. What do you refuse to build? An agency with no refusals is an agency with no standards. The right answer names something: "we won't build a chatbot that answers from documents it can't cite," for example.

9. How do you handle my data? Where does it live? Who can see it? What happens when the engagement ends? Vague answers here are a red flag — your data is the asset you're handing over.

Fit: will this actually work between two companies?

10. Who do I actually talk to? If every conversation routes through a salesperson, the build will feel like that too. You want a direct line to the people doing the work.

11. What does success look like in 30 days? A serious agency can describe the first milestone concretely — what exists, what you can touch, what's measured. "We'll start with discovery" is a stall, not a plan.

12. What did your last client say when it went wrong? Ask for an honest war story. Every real project has one. How they handled theirs predicts how they'll handle yours.

How we answer these at SILKDEV

Since we're asking you to hold us to the same bar: we build the SILKDEV product family in-house (knowledge infrastructure, AI-native accounting, and more) — the portfolio is public and live. We quote fixed prices from your brief, we run projects in a visible client portal with an AI project representative, and we ship with evidence rather than promises. We also refuse work: we don't build citation-less knowledge bots, and we don't sell "AI" as a magic layer over a broken process.

FAQ

Should I choose a local agency or a remote one?

For AI work, location matters less than accountability. A remote agency with a visible process and a fixed price can outperform a local one with a black-box workflow. What matters: can you see progress, and can you reach the people building?

How do I evaluate an agency's portfolio?

Use it. Open the product, do a real task, look at the edges — error states, empty states, load times. A portfolio you can't use is a brochure.

What's a reasonable timeline for an AI project?

A focused feature: 2–6 weeks. An agent system with integrations and guardrails: 1–3 months. Anyone promising production AI in a week is either tiny-scoped or selling you a demo.

How much should I budget?

Small automations: low thousands. Production features: tens of thousands. Agent platforms: more. Budget for the outcome, not the technology — and get a fixed price before you commit to a number.

The short version

Twelve questions, four buckets: capability, process, handling, fit. If an agency can show production work, name a fixed price, describe a 30-day milestone, and answer who owns quality — you've found a real one. Tell us what you're building — we'll scope it and get back to you within 24 hours.