Stop Asking “Can AI Do This?” Ask “What Happens If It’s Wrong?”

AI can write a meeting summary, compare two specifications, propose a test plan, and produce a confident answer to a technical question. Those outputs may look equally polished. Their consequences are not equal.

Medical-device engineering taught me to ask what an error could do before deciding how much evidence is enough. That habit carries into AI-assisted work: a wrong meeting-summary detail and a wrong technical conclusion may both sound confident, but they require very different review.

In engineering and program work, I find “Can AI do this?” to be the wrong first question. The better one is “What happens if it is wrong?” A weak brainstorm costs a few minutes. An incorrect owner or date can disrupt a program. A mistaken conclusion about product safety or clinical use can be far more consequential. One blanket rule for all three tasks is unlikely to help.

Classify the task before choosing the tool

I use three dimensions when deciding how much AI assistance to allow:

1. Consequence: What could happen if the output is wrong and someone acts on it?
2. Detectability: Can a competent reviewer find the error before it matters?
3. Reversibility: Can the team correct the outcome cheaply, or does the decision create a lasting commitment?

The result is a task-risk matrix, not a score that grants permission. A task with low consequence may still involve confidential information that should not enter an unapproved tool. Conversely, a high-stakes task can still use AI for a narrow support step, such as locating passages for a qualified person to verify.

Task	Typical consequence if wrong	Appropriate AI role	Human check before use
Brainstorm alternative meeting titles	Low and easily reversible	Generate options	Select for clarity and tone
Extract action items from notes	Moderate; an owner or date may be misreported	Structure the source	Check each action against notes and confirm ownership
Summarize a supplier concern for leadership	Potentially high; wording may change a decision	Draft from verified facts	Validate impact, uncertainty, escalation trigger, and ask
Decide whether a safety-critical requirement is satisfied	High and potentially hard to reverse	Assist with locating evidence, if approved	Qualified owner evaluates evidence and signs off through the established process

This is why the most impressive-looking AI demo is not necessarily the best first use case. I would start where errors are visible and inexpensive, then expand only when the team knows how it will review the output.

Separate finding information from making a decision

Imagine a model reviews a design-history folder and says, “All verification evidence is complete.” It may have found many documents. That does not establish that the right tests were run, that deviations were dispositioned, or that the evidence applies to the current design revision.

A safer task is narrower: “List the documents that appear to address requirement R-14, identify their revisions, and quote the passages that support your list. Mark missing or ambiguous evidence.” A human can then decide whether the evidence is sufficient. The AI has helped with retrieval and organization; it has not inherited the engineer’s or quality function’s authority.

The NIST AI Risk Management Framework is voluntary guidance for managing AI risks in context. Its generative AI profile addresses risks that include confidently wrong output. Neither document is a stamp of approval for a particular prompt. I read them as a reason to define the use case, test the output, and keep accountability explicit.

Make the review proportional

For a low-risk draft, review may mean a quick read. For a program update, I would check numbers, sources, owners, dates, and whether proposals were accidentally promoted into decisions. For a regulated or safety-relevant conclusion, the review belongs inside the applicable engineering and quality process. “A human looked at it” is not enough if that person lacked the evidence, time, or authority to evaluate it.

Two questions expose weak review plans:

• Could the reviewer independently detect a plausible error? If not, the review may be ceremonial.
• Would the team know who made the final call? If the answer is “the AI suggested it,” ownership is unclear.

I also separate output risk from data risk. A harmless writing task can still be inappropriate for a tool if it requires uploading confidential client, employer, patient, or proprietary material. The organization’s approved tools and data rules are a gate before the task-risk matrix even begins.

Use AI where it improves the system

My best use cases tend to have a clear source and a clear handoff: extract open questions from notes, compare document versions, outline alternative communications, or flag missing fields. The model does mechanical information work. A responsible person verifies what matters and decides what happens next.

That approach complements my earlier piece on using AI to manage complex engineering programs. The objective is not to keep AI away from serious work. It is to assign it a role that the team can explain, inspect, and defend.

Before putting AI into a workflow, write down the task, the possible failure, the evidence required, and the decision owner. If those four lines are hard to fill in, the workflow is not ready to scale.

Before adopting a new AI workflow, measure the review burden as well as the drafting speed. Is AI Actually Saving You Time? offers a practical way to do that.

If you are mapping practical AI uses in a technical team, send me a note. I am interested in where a small, well-reviewed use case removes real work without blurring accountability.

Keep reading

• Not Everything Needs AI: What Building a Form-Filler Taught Me About When Software Still Wins