AI clearly offers benefits. But when it comes to security, it also brings its own risks.
Artificial intelligence is often described as a “black box”. We can see what we put in, and what comes out. But not how the model comes to its results.
And, unlike conventional software, large language models are non-deterministic. The same inputs can produce different results.
This makes it hard to secure AI systems, and to assure their users that they are secure.
There is already growing evidence that malicious actors are using AI to find vulnerabilities, carry out reconnaissance, and fine-tune their attacks.
But the risks posed by AI systems themselves could be even greater.
Our guest this week has set out to secure AI. His approach: red team testing methods that take into account both the nature of AI, and the unique risks it poses.
AI systems face many of the same risks and threats as conventional software, including vulnerabilities in code, and data leakage.
And there are new threats too. These include prompt injection, manipulation of results, and even convincing AI agents to carry out actions they should not.
So how do we make sure AI is secure, when we can’t see inside the models, or see the data they are trained on?
Dr Peter Garraghan is a professor at Lancaster University, and founder and CEO at Mindgard. His firm that specialises in continuous, red team testing of AI.
He argues that investments in AI must be matched by investments in security capabilities, as well as a better understanding by businesses of what, exactly, they want their AI tools to do.
Interview by Stephen Pritchard.
Featured image by Stefan Schweihofer from Pixabay
