Q&A
Securing AI Models from Attacks
Cybersecurity expert Chen Shiri breaks down the challenges of safeguarding large language models and some steps you can take to secure your organizational data.
As threat actors increase their attacks on large language models, securing enterprise AI against growing attacks has become a critical challenge for cybersecurity professionals. According to a recent IBM study, 55 percent of telecommunications executives reported network security breaches within the past year, highlighting the urgent need for robust security measures as AI adoption grows.
However, only 42 percent of these leaders view network security as a top challenge, revealing a concerning gap in priorities as companies increasingly integrate AI and cloud technologies into their operations.
Chen Shiri, a cybersecurity researcher at Accenture Security, is at the forefront of addressing these challenges. In a Q&A with Redmondmag, Shiri delves into the complexities of defending AI systems, particularly those operating in black-box environments where model architecture and parameters remain hidden. He explains why traditional cybersecurity approaches often fall short against sophisticated AI threats such as model enumeration, model stealing and adversarial attacks.
Shiri will expand on these insights during his upcoming session, "NeuroInvasion: Penetrating the Core of Artificial Intelligence", at the virtual Cybersecurity & Ransomware Live! (May 13-15). This intermediate-to-advanced presentation will introduce innovative techniques to exploit black-box AI models, including the use of substitute models to apply white-box adversarial strategies.
For security professionals aiming to stay ahead of adversaries, this conversation offers a preview of cutting-edge methodologies and the critical skills needed to fortify AI models effectively. Don't miss this opportunity to learn how to bridge the gap between AI innovation and cybersecurity resilience. Register today!
Redmondmag: Why is securing AI models against these threats so challenging?
Shiri: AI security is tough because black-box models don't reveal their architecture or parameters, making it hard to defend against attacks. Adversaries can exploit input-output behaviors, adversarial examples, and cloud/API vulnerabilities without direct access to the model. Attackers use model enumeration, model stealing and adversarial inputs to manipulate AI decisions, making traditional cybersecurity defenses ineffective.
What is model enumeration, and why is it critical in AI attacks?
Model enumeration is a probing technique used to infer an AI model's architecture by analyzing how it responds to specific inputs. Attackers use prompt engineering and output comparison (input-output testing, adversarial prompts, etc.), response time analysis and statistical fingerprinting to uncover hidden details, allowing them to craft better attacks. Once the structure is identified, they can apply white-box attack methods on a replicated model and transfer those adversarial inputs to the real system.
What are some of the most alarming AI vulnerabilities you've uncovered?
(Only from the things that I'm allowed to talk about…)
- Model Stealing: Attackers can replicate black-box AI models by querying APIs, extracting patterns, and training a substitute model to bypass security controls.
- Cloud AI Exploits: Poorly secured AI services on AWS, Azure, or Google Cloud are vulnerable to API injection, SSRF attacks and privilege escalation.
- Adversarial Attacks: Minor changes in inputs can completely alter AI predictions in security-sensitive areas like facial recognition, self-driving cars and fraud detection.
What skills should security professionals develop for AI penetration testing?
- Machine Learning Basics: Understanding neural networks, transformers, and adversarial training.
- Adversarial AI Attacks – Techniques like FGSM, DeepFool, CW Attack, and model enumeration.
- Cybersecurity & API Security – Protecting AI APIs from model leaks and abuse.
- Cloud & Infrastructure Security – AI services are often deployed in cloud environments, making container security and API protection critical.
What are the biggest mistakes companies make when integrating AI into security?
- Assuming AI is Secure by Default. Many ignore adversarial testing, making AI an easy target.
- Exposed AI APIs. Poor API security leads to model theft, enumeration attacks, and injection risks.
- Ignoring Cloud Security. AI in AWS, Azure, or GCP can be exploited via cloud misconfigurations.
- Lack of Monitoring. AI models drift over time, and without continuous monitoring, they can be poisoned or manipulated.