As artificial intelligence systems move rapidly from experimentation into production, many organizations are discovering that traditional testing fails to surface the most serious risks. According to Arun Kumar Elengovan, Director of Engineering Security at Okta and Advisory Council Member at Products That Count, the most consequential AI vulnerabilities often emerge only after models reach real users.
In a recent virtual speaker session hosted by Products That Count, Elengovan delivered a technical briefing titled "Building Guardrails for Responsible AI: Starts with Breaking AI." The session focused on why AI failures are rarely accidental and why effective safeguards must begin with understanding how systems are deliberately exploited.
Why AI Fails Differently Than Traditional Software
Elengovan opened by explaining how AI systems fail in ways fundamentally different from traditional software. In conventional systems, code and data are clearly separated. AI systems, especially large language models, process instructions, user input, and retrieved content as a single stream of tokens. This architectural reality makes it impossible for models to reliably distinguish authority from data.
This "data-code boundary problem," as Elengovan described it, is not a defect that can be patched. It is intrinsic to how modern AI systems operate. As a result, entire categories of attacks—prompt injection, indirect instruction execution, and multiturn manipulation remain persistent across models.
From Architecture to Exploitation
To demonstrate the real-world impact of these limitations, Elengovan highlighted recent vulnerabilities where AI-generated outputs became vectors for phishing, browser-side execution, and unauthorized data exfiltration. In these cases, the AI system itself was not malicious, but its outputs crossed execution boundaries that traditional security models were never designed to defend.
He also discussed advanced attack techniques such as Crescendo and Skeleton Key attacks, which use multi-turn conversations to gradually erode safety controls. These techniques have proven effective across multiple major commercial models, reinforcing that one-time prompt filtering is insufficient for production environments.
Why Social Engineering Works on AI
One of the most compelling segments of the session examined why AI systems are particularly susceptible to social engineering. Large language models are trained to be helpful, compliant, and conversational. Reinforcement learning rewards cooperation, not skepticism.
As a result, emotional pressure, authority cues, and narrative framing that would raise concern for a human often succeed against AI systems. Elengovan illustrated how guilt, urgency, flattery, and fictional storytelling can all weaken guardrails if systems are not explicitly designed to resist these patterns.
"The model is not failing," he noted. "It's doing exactly what it was trained to do."
Proactive Stress Testing as the Foundation of Responsible AI
Rather than treating AI security as a compliance checklist or a post-deployment concern, Elengovan emphasized the importance of systematically stress testing AI systems before they reach users. By intentionally probing models for failure modes, manipulation vectors, and unintended behaviors, organizations gain a realistic understanding of how their systems behave under pressure.
This approach shifts Responsible AI from theory into practice. Instead of assuming safeguards are effective, teams validate them through structured evaluation, adversarial exploration, and continuous reassessment. These exercises reveal architectural weaknesses that would otherwise remain hidden until exploited in production.
Elengovan stressed that principles such as fairness, transparency, and accountability only become meaningful when paired with measurable evidence and repeatable validation processes. Without this rigor, Responsible AI efforts often remain aspirational rather than operational, leaving organizations exposed to avoidable risk.
Leadership at the Intersection of AI and Security
Elengovan's presentation reflects a broader shift in how leading organizations approach AI safety. As a Fellow of the British Computer Society (BCS) and IETE, IEEE Senior Member, Gartner Information Security Ambassador, Distinguished Fellow of the Soft Computing Research Society, Head judge for multiple hackathons such as DreamwareHack, and Vice-Chair of an ACM professional chapter, his work bridges deep technical expertise and industry leadership.
By reframing AI security as an architectural challenge rather than a tooling problem, Elengovan is helping product leaders understand what it truly takes to build trustworthy AI systems at scale. As AI adoption accelerates, his message is clear: responsible AI begins not with blind optimism, but with a rigorous understanding of how systems fail.
