AI Security
Check the AI Output Before It Reaches a Business System
Build a review around the point where generated content becomes a database value, rendered page, or business action, with repeatable rejection tests.
Top One Percent AI · Practical guidance with linked sources.
Consider two hypothetical failures. An AI assistant proposes a customer-record update. Its response is well written and uses valid JSON, a structured data format. The integration accepts it. The wrong customer record still changes. In another workflow, a helpful summary contains markup that a dashboard interprets as active content. Both examples illustrate why checking the prompt and checking the answer for accuracy leave an important gap. The receiving system needs its own rules for what generated output may become.
For an organizer planning an AI security session, this boundary makes a useful, concrete subject. Ask the speaker to follow one generated result from the model into a browser, database, or business application, and show where it can be rejected.
Start with the destination
OWASP's guidance on improper output handling describes risks arising when model output reaches downstream components without sufficient validation and handling. Its examples include generated commands entering a shell and generated content being interpreted by a browser. Its recommendations include validating model responses and using encoding appropriate to the destination.
A practical review should therefore begin with an inventory of destinations. A paragraph displayed as plain text, a database field, and a proposed executable command need different treatment. Calling everything an “AI response” hides the difference.
For each destination, record the accepted data, the transformation applied to it, the business operation it can trigger, and the component that rejects an invalid result. Name an owner for that component. This is an implementation recommendation, not a claim that one checklist guarantees security.
Separate shape from meaning
Structured output is helpful because applications can require a predictable format. However, OpenAI's Structured Outputs documentation explicitly notes that structured outputs can still contain mistakes. A schema does not establish that a proposed business action is correct.
Consider an illustrative order-adjustment workflow. The output contains an order identifier, a replacement delivery date, and a reason. Format checks can reject missing fields or an invalid date. Business checks must separately establish that the order exists, belongs to the relevant customer, remains changeable, and permits that delivery date. The application should look up these facts from authoritative records rather than accept the model's assertions about them.
Treat missing evidence as a defined outcome. A system that silently invents a default customer, substitutes today's date, or strips a failed field can turn a visible problem into an apparently successful transaction. Route the proposed change for correction or review instead.
Keep generated values away from executable structure
For database access, ask the model for constrained values needed by an application-owned operation rather than unrestricted SQL, a database query language. OWASP's SQL Injection Prevention Cheat Sheet recommends parameterized queries for data values. It also distinguishes structural choices, such as table names, that require fixed choices or an allow-list rather than ordinary parameter binding. Parameterization does not decide which account the caller is entitled to access.
For displayed content, make plain text the default when rich rendering is unnecessary. Where formatting is required, have the application enforce the permitted elements and apply the protections appropriate to that renderer. Avoid treating an instruction in a prompt such as “return safe HTML” as the control that protects the browser.
The same design question applies to exports, file names, and requests through an application programming interface (API). Decide which parts the application owns and which bounded values the model may propose. Give the receiving component the final say.
Test the handoff directly
An output-validation test does not need a model to produce the bad result on demand. In a test environment, substitute prepared outputs at the model boundary and verify the receiving component's behavior. This makes the exercise repeatable even when model responses vary.
Include cases that are syntactically valid but wrong for the workflow. Try an unknown record identifier, a quantity outside an agreed limit, a prohibited state transition, an extra field, and an otherwise acceptable result associated with the wrong customer. Include incomplete responses and explicit refusals so they cannot be mistaken for completed work.
Write down the expected observable result for every case. “Rejected with no record changed” is stronger than “the system should notice.” Check that rejection produces a useful reason without exposing sensitive content. Capture enough identifiers and validation outcomes to investigate failures, with access and retention appropriate to the data.
Turn the session into a review artifact
Give workshop groups one fictional workflow and a blank handoff record. Ask them to specify the destination, permitted fields, authoritative lookups, rejection behavior, and five test cases. Have another group play the receiving application and challenge every assumption.
A strong speaker brief asks for a safe demonstration of one accepted result, one format failure, and one business-rule failure. The takeaway should be a review record participants can apply to a real integration. The session succeeds when attendees can identify the exact check that stands between generated content and a consequential system change.