Dummies save lives. Specifically, crash dummies. Those plastic bodies bolted into car seats are actually sophisticated technology that influences how we build the safety features we\u2019ve come to expect in vehicle design.<\/p>\n
Just like the once popular crash test dummy commercials said,<\/a> you can learn a lot from a dummy.\u00a0 Thanks to them, seatbelts improved, airbags were added, and cars started to crumple in the right places instead of crushing passengers.<\/p>\n Just like a crash dummy reveals hidden dangers before cars hit the road, a well-built AI product pilot uncovers potential flaws before scaling.<\/p>\n Once an organization aligns a new AI offering around customer problems and company goals, you\u2019re ready to assemble a pilot that focuses on answering two critical questions:<\/p>\n \u25cf Phase 1: Will the product or feature work? Pilots should balance between a sandbox for testing the viability of a product or feature idea and factors that simulate the real world.<\/p>\n In other words, it needs to have enough guardrails to give it a chance and a controlled environment so your teams can closely observe, document and analyze what’s happening.<\/p>\n Practically, this means carefully choosing participants who truly represent your product\u2019s real users and diverse use cases, like:<\/p>\n \u25cf A mixture of active and daily engaged users and infrequent users. During the pilot phase, a product leader\u2019s primary role is asking, \u201cAre we actually solving the intended problem in a meaningful way?\u201d<\/p>\n You\u2019ll also:<\/p>\n \u25cf Help your team prioritize features based on value, complexity and feasibility by identifying trade-off decisions between must-have and nice-to-have capabilities.<\/p>\n \u25cf Continuously manage expectations with internal stakeholders by transparently sharing progress and challenges. And remember, when AI and ML is involved, there is a higher level of education needed. That makes this step more challenging than it is for a typical project.<\/p>\n \u25cf Advocate for a mindset of experimentation, learning and adaptation rather than an immediate answer to \u201cDid this succeed or fail?<\/p>\n There\u2019s a temptation during pilots to create an environment designed for success and then get prematurely excited when it does succeed. That\u2019s just being human, and it is exciting. However, the pilot should include a second phase after viability: stress testing. A pilot isn\u2019t just feasibility; it\u2019s your first risk assessment.<\/p>\n Imagine if we put our crash dummies in a car traveling at safe speeds with no accidents as proof of safe design. It wouldn\u2019t be too terribly helpful. Instead, we put them in the worst possible situations to understand risk.<\/p>\n The pilot should first determine if the idea works in a realistic environment, and then you should ask, \u201cWhat if everything that could go wrong did go wrong?<\/p>\n Hackers for Hire: <\/strong>Google has a dedicated team for this kind of work: the Red Team. It was created in 2016 to improve security.<\/p>\n As Google explains,<\/a> \u201cThe main point is for the red team to emulate the same behaviors a real threat actor would use in a real-world scenario to gain a foothold and remain undetected. Identifying vulnerabilities and other limitations can help us zero in on the specific technical elements that make attacks successful, so we can extract them, study them, and implement solutions that make them less effective.\u201d<\/p>\n How do we know what could go wrong? You could make your best guess about the highest risks, but a better answer might come from a pre-mortem with internal teams like security, ops, and support teams. You can even us AI for your pre-mortem; it is a great thought-partner and can be especially helpful in a risk-mitigation session<\/p>\n Ask them to imagine the project failing when it moved from 1,000 users or queries a day to 10,000. Then, have them theorize what might have caused the failure. They might find areas of concern based on their expertise, like:<\/p>\n 1. Systems that talk to each other only in the lab<\/strong><\/em> 2. Performance that crumbles under real traffic<\/strong><\/em> 3. Rules and people you didn\u2019t meet in pilot mode<\/strong><\/em> It\u2019s possible none of this showed up in the first phase of the pilot.<\/p>\n Consider creating a separate sandbox, but instead of building it to test if the idea will work, try to make it fail. Identify your biggest risk concerns and then simulate them with the same focus on what the real world could look like, but in unfavorable conditions. 1. What if the model is retrained or updated?<\/strong><\/em> 2. What if your model, data pipeline, or dependencies don\u2019t update together?<\/strong><\/em> 3. What if someone intentionally misuses the system?<\/strong><\/em> 4. What if the product breaks quietly and no one notices?<\/strong><\/em> Goal:<\/strong> Test how much poor-quality data your model can handle before performance drops below acceptable levels, and identify which types of messiness are most harmful.<\/p>\n Step 1:<\/strong> Break down what low-quality data looks like in your context: missing values, duplicates, inconsistent formatting, typos, slang, irrelevant data.<\/p>\n Step 2:<\/strong> Start with your clean dataset. Then gradually introduce messiness in controlled increments: 5% noise, 10% noise, 25%, 50%, 75%, etc.<\/p>\n Step 3:<\/strong> Define the \u201cbreaking point\u201d where the model is no longer usable.<\/p>\n Step 4:<\/strong> Document the total percent of messiness it can handle before performance dips significantly, and\/or the type of error it\u2019s most sensitive to (e.g., label errors vs. missing values)<\/p>\n Advanced Stress Testing Thought Exercises<\/strong><\/p>\n If you want to do more in-depth planning for the future, you can facilitate thought experiments like the ones Jane McGonigal explains in her book Imaginable<\/em>. She encourages paying attention to signals, trends, emerging tech, or small changes in behavior that might grow over time.<\/p>\n It\u2019s not just creative thinking. It\u2019s a structured way to stress-test your assumptions. McGonigal talks a lot about urgent optimism, or the belief that you can make a difference in the future, and her exercises help you feel more prepared.<\/p>\n For example: \u25cf What happens if a vendor you\u2019ve partnered with on the project shuts down their business? Do you lose your data? Do you lose your model? Do you have a contingency plan?<\/p>\n \u25cf What is one law or regulation that could change everything? How could it make your product better, worse, or illegal?<\/p>\n Crash dummies aren\u2019t valuable because they survive crashes; it\u2019s because they don\u2019t.<\/p>\n If your AI pilot is your crash dummy, the goal isn\u2019t keeping the product or feature safe in the sandbox. It\u2019s knowing exactly where you have risk and building a plan to respond. It\u2019s this kind of work that will earn trust not just with leadership but also with your most skeptical customers.<\/p>\n It\u2019s also a story of iterative, intentional cycles of controlled failure and targeted improvement.<\/p>\n AI isn\u2019t predictable; pilots like these are designed to manage complexity, and the approach isn\u2019t new. In the early 2000s, IBM\u2019s Director of the Cynefin Centre, Dave Snowden, introduced what he called \u00a0safe-to-fail probes<\/a> to cope with the inherent uncertainty of adaptive systems. It was used with clients like the UK Ministry of Defense to improve life fire exercises for soldiers, a large Asian-Pacific bank for customer retention and a hospital to find strategies to reduce nurse workload.<\/p>\n His approach was straightforward:<\/p>\n \u25cf Run many small probes. Treat your AI pilot the same way. Break it on purpose, learn fast, adapt, and then you\u2019re better prepared to start scaling.<\/p>\n Additional AI Resources for Product Professionals:<\/strong><\/p>\n Stop Chasing AI Hype: Start Solving Market Problems<\/a><\/p>\n Leveraging AI to Improve Market Research<\/a> (Podcast)<\/p>\n![\"Phases](\"https:\/\/www.pragmaticinstitute.com\/resources\/wp-content\/uploads\/sites\/6\/2025\/06\/Test_Crash_Art1-300x158.png\")
\n\u25cf Phase 2: What\u2019s the risk?<\/p>\nWill the product or feature work?<\/h2>\n
\n\u25cf Data sets with a little noise, rather than sanitized, overly simplified test sets.
\n\u25cf Customers with varying levels of technological comfort or familiarity with your product.<\/p>\nWhat\u2019s the risk<\/h2>\n
![\"A](\"https:\/\/www.pragmaticinstitute.com\/resources\/wp-content\/uploads\/sites\/6\/2025\/06\/Test_Crash_Art3-300x300.png\")
<\/p>\nIdentifying Product Risk with a Pre-Mortem<\/h2>\n
\nThe pilot pulls data from one clean spreadsheet. In production, it must sync with four live systems, each on a different update schedule.<\/p>\n
\nA model that answers 200 queries a day is fine on a single server. At 20,000 queries, it needs load\u2011balancing, caching, and new alert rules.<\/p>\n
\nSecurity asks for a formal pen test. Compliance wants an audit trail. The help desk needs a script for angry callers.<\/p>\nPossible Crash Tests<\/h2>\n
![\"\"](\"https:\/\/www.pragmaticinstitute.com\/resources\/wp-content\/uploads\/sites\/6\/2025\/06\/Test_Crash_Art2-300x300.png\")
<\/p>\n
\nDo key behaviors change? Does it forget critical features or reintroduce bugs? Check what happens to historical consistency and user expectations.<\/p>\n
\nIntentionally mismatch versions and see where breakdowns occur in processing, logging, or UI.<\/p>\n
\nTry prompt injection, adversarial inputs, or permission abuse. Can users manipulate outputs, flood content, or access restricted features?<\/p>\n
\nDisable logging, reduce alert thresholds, and see how long it takes for someone to catch and respond to errors.<\/p>\nExample: What a Test Design Might Look Like<\/h3>\n
\n\u25cf What\u2019s one thing your team assumes will stay the same? Now, imagine the opposite happens. What breaks in your product, your process, or your customer relationship?<\/p>\nThe Crash Dummy\u2019s Final Lesson<\/h2>\n
\n\u25cf Watch what fails and why.
\n\u25cf Amplify what works, fix or discard what doesn\u2019t.<\/p>\n