I recently went through a small research, asking – What is the AI equivalent of cybersecurity’s Red Team / Blue Team model?
The only answer I could find seems to be: there isn’t one universally accepted model yet.
But there are several emerging roles that make the AI security landscape much easier to understand.
I’m not approaching this as an AI security practitioner. What interested me was finding a high-level mental model that makes sense from a product and business perspective.
Here’s where I landed.
Red Team: break it.
- Try prompt injections. Jailbreak it. Attempt to extract data. Manipulate it. Find adversarial inputs. Make it hallucinate or circumvent business rules.
- The question: Where are the weaknesses?
Evaluation: measure it.
- How accurate is it? How often does it hallucinate? Is it consistent? Biased? Does reasoning hold up? Does performance regress after a model change?
- The question: How good is it at what we actually need it to do?
Safety and guardrails: constrain it.
- Add filtering, access controls, grounding, output validation and human oversight.
- The question: How do we reduce undesirable behavior?
Governance: decide.
- Is this use case appropriate? Can this data be used? Who owns the decision? Can the output be explained? What happens when the model gets something wrong?
- The question: Should we be doing this in the first place?
Monitoring: watch reality.
- Because passing every pre-production test doesn’t guarantee that the system will behave the same way once deployed.
- The question: What’s actually happening in production?
So my simplified version is: Attack → Measure → Govern → Monitor
And surround the whole thing with appropriate defenses.
What I like about this framework is that it makes “AI security” feel less like one enormous technical discipline and more like a sequence of understandable business questions.
It also exposes why saying “we tested the AI” isn’t particularly informative.
Tested for what?
Security vulnerabilities?
Accuracy?
Bias?
Business appropriateness?
Regulatory concerns?
Production drift?
Those are different questions, and potentially different responsibilities.
The terminology around AI security will almost certainly continue changing. But the underlying jobs are becoming easier to see.
Don’t just ask whether an AI system is safe or secure. Ask who is responsible for finding out how it fails, and what happens when it does.
Leave a Reply