Most AI development firms can build a compelling demo. The hard part is building something that still works six months after handoff.
RAND Corporation's 2025 analysis found that 80% of AI projects fail to deliver their intended business value. A third of those never even reach production. S&P Global reported that 42% of companies scrapped at least one AI initiative in 2025 - up from 17% the year before.
The pattern behind these failures isn't usually bad technology. It's mismatched partnerships: companies hiring demo shops when they needed production teams, or enterprise consultancies when they needed scrappy builders.
If you're evaluating AI development partners right now, here are the seven questions that reveal what you're actually buying.
1. "Show me something you built that's running in production today."
This is the single most effective filter question. Not a case study from 2023. Not a prototype they built for a conference talk. A system that is handling real data, serving real users, and being monitored right now.
As Rocket Farm Studios put it in their 2026 buyer's guide: "'Built a chatbot' tells you nothing. 'Deployed a claims-processing AI agent that reduced manual review time by 62% across 14,000 monthly transactions' tells you everything."
What good looks like: They can walk you through a live monitoring dashboard. They can describe a recent production incident and how they resolved it. They know the system's current error rate without looking it up.
Red flag: Everything they reference is a prototype, a proof-of-concept, or a project that ended at delivery. No mention of what happened after launch.
2. "What happens to this system after you hand it off?"
McKinsey's State of AI research (November 2025) found that only 5.5% of companies qualify as AI high performers - organizations where AI contributes meaningfully to EBIT. One defining behavior of that group: they treat AI development partnerships as capability-building programs, not one-time delivery contracts.
The question reveals whether your potential partner has thought about the full lifecycle or just the build phase. A system that degrades within 90 days because nobody is monitoring for data drift isn't a successful delivery - it's an expensive experiment.
What good looks like: They describe a structured handoff process. They can explain what your team needs to know to maintain and extend the system independently. They have documentation standards and knowledge-transfer checkpoints built into their delivery model.
Red flag: "We offer ongoing support contracts" is the only answer. No mention of what gets transferred, validated, or documented. The implicit model is perpetual dependency.
3. "Walk me through how you'd monitor this system for drift and degradation."
Gartner's May 2026 research predicts that only 40% of organizations deploying AI will use proper observability to monitor model performance by 2028. That means the majority of production AI systems today are flying blind.
This question tests whether a partner treats monitoring as a core architectural concern or a "Phase 2" nice-to-have. If they can't describe their approach to drift detection, performance dashboards, and alerting before a contract is signed, they haven't built enough production systems to know it matters.
What good looks like: They name specific monitoring approaches - drift detection methods, quality metrics they track, alerting thresholds. They can describe a situation where monitoring caught a problem before users did.
Red flag: Monitoring is described as "included" without specifics. Or worse: it's positioned as a separate add-on after the main build is complete.
4. "When a project goes wrong, what does that look like? Give me a real example."
Every experienced team has war stories. The partner who has never lost a sprint, never had a model underperform, and never needed to pivot mid-engagement is either lying or hasn't built enough to encounter real production complexity.
This question accomplishes two things: it tests self-awareness, and it reveals how the team handles adversity. The answer tells you more about working with them than any polished case study.
What good looks like: They describe a specific project that went sideways, what they learned, and what they'd do differently. They're specific about the failure mode - not vague about "communication challenges."
Red flag: They deflect. Every project was a success. Problems were always caused by the client's data or the client's timeline.
5. "What would you tell us NOT to build?"
A development shop that says yes to everything is optimizing for revenue, not outcomes. The best partners will push back when a proposed approach is wrong - when the use case doesn't justify the complexity, when a fine-tuned model isn't better than a well-prompted API call, or when the data foundation isn't ready.
McKinsey's high-performer research found that organizations achieving real AI ROI are 3× more likely to have set outcome-based objectives tied to business KPIs before building. Partners who enable that discipline will tell you when something isn't worth building.
What good looks like: They have a clear framework for when AI is and isn't the right solution. They can name a situation where they talked a client out of a more complex (and more expensive) approach in favor of something simpler that delivered the same outcome.
Red flag: They say yes to your entire wish list in the first meeting without asking hard questions about priority, data readiness, or expected ROI.
6. "Who exactly will work on our project?"
This is where boutique firms and large agencies diverge sharply. The people in the sales meeting are often not the people doing the work. The senior architect who impressed you in the pitch may be spread across six accounts or may not touch your project at all.
What good looks like: They introduce the actual engineers and technical leads who will be assigned to your work. Those people can speak technically about relevant past projects. The team is stable - not a rotating cast assembled per-project from a bench.
Red flag: You're told you'll get "a senior team" without names. The proposal focuses on the firm's credentials rather than the specific humans doing the work. There's a separate "delivery team" you haven't met.
7. "How do you define 'done' - and how do you define 'working'?"
This question exposes the gap between technical completion and business value. A partner oriented toward real outcomes will define success in terms you care about: reduction in processing time, accuracy improvement that moves a business metric, cost savings that show up in your P&L.
A partner oriented toward delivery milestones will define success in terms they control: model deployed, endpoints live, documentation delivered.
Both are necessary. But if the conversation stops at technical deliverables and never connects to business outcomes, you're buying a project, not a result.
What good looks like: They ask about your business KPIs before proposing technical metrics. They can explain how they've tied model performance to business outcomes in past engagements. Their SOW includes success criteria that matter to your business, not just their engineering team.
Red flag: Success is defined exclusively as "delivered on time and on budget." No mention of whether the system actually achieved what it was built to achieve.
The Pattern Behind These Questions
Notice what these seven questions have in common: none of them are about technology. They don't ask which LLM a partner prefers, which cloud provider they use, or whether they've worked with your industry's specific data format.
That's intentional. Technology questions have easy answers. A partner can learn your industry's data constraints in a week. They can't learn production discipline, intellectual honesty, or outcome orientation in a week. Those are either embedded in how they work or they aren't.
The technology matters, but it's table stakes. The questions above test for the things that actually predict whether your AI investment will survive contact with reality.
How to Use This Framework
If you're early-stage (first AI project, unclear scope): Focus on questions 4, 5, and 7. You need a partner who will help you figure out what to build - not just build whatever you ask for.
If you're scaling (pilot worked, now expanding): Focus on questions 1, 2, and 3. You need a partner whose systems survive at scale and whose handoff process actually works.
If you're replacing a partner (previous engagement failed): Focus on questions 1, 4, and 6. You need to verify production experience, learn from what went wrong last time, and ensure the people doing the work are the people you're evaluating.
At Apptitude, we build AI agents and intelligent software for founders and operators who need production systems, not science projects. If you're evaluating development partners and want to have this conversation, reach out - we'll give you honest answers to all seven questions.