Between Trust and Responsibility: AI Security Reimagined
Artificial intelligence (AI), the security of AI systems and the use of AI in security are no longer a thing of the future – they are our present. And they have long been an integral part of our daily work to improve IT security. At the same time, they bring with them a new quality of risks that we in the security industry must take very seriously.
From Bach to Artificial Intelligence: a Journey Through Time
My first encounter with AI was a long time ago. In 1979, a friend of mine spent every spare minute reading a thick white book called “Gödel, Escher, Bach”. As a musician, I was initially only interested in the aspect relating to Johann Sebastian Bach. Unfortunately, it didn’t help me much in my attempts to play “The Well-Tempered Clavier” and its fugues. But I did learn about AI.
In the book, author Douglas R. Hofstadter describes how complex, intelligent behavior can arise from astonishingly simple systems. The idea: self-referential loop structures that create levels of meaning – whether in logical proofs, drawings, or musical compositions.
Bach’s fugues repeatedly contain melodies that contain themselves and are played simultaneously in variations, creating a new musical level in which the individual melody seems to disappear again and again, but is actually always there. So, it’s a bit like what we experience with large language models and generative AI: the individual disappears into a new, larger whole.
When complex structures generate meaning at a higher level and we can then map this using digital tools, we call the whole thing artificial intelligence. Hofstadter refers to these as rule-based systems. Our current AI systems do this in the form of neural networks, which produce seemingly “intelligent” outputs through the interaction of billions of parameters. Similar to flocks of birds, ant colonies, or the stock market, emergent behavior arises: we use systems that we no longer fully understand, but whose results seem plausible and useful enough to us to use them.
The Balancing Act Between Usefulness and Control
After decades of development, the use of artificial intelligence has become commonplace in recent years. Whether we have already largely exhausted its actual capabilities or are still at the very beginning of an exponential curve remains to be seen. We are on the critical path from simple dialogue functions to semi-autonomous or even autonomous systems. Here, technical efficiency (e.g., response time or loss tolerance) naturally conflicts with human qualities such as judgment, responsibility, and the ability to make reasoned decisions. And because we are making AI systems increasingly powerful, the question becomes more urgent: How secure, vulnerable, and trustworthy are they really?
Trust
Just as with our partners, colleagues, and people in general, we cannot fully understand the internal processes of AI. This not only makes AI difficult to verify, but also particularly vulnerable to targeted manipulation, whether through adversarial attacks or subtle input distortions. But not using AI is obviously not a solution either. There is no way around it: we have to deal with it. We can only establish trustworthy tools and processes that protect us well enough.
It is in the nature of things that we have to be content with statistical probabilities instead of provable truths that we can understand. In practice, this usually works well – but not always. Where trust is based on habit rather than understanding, there is no basis for control in an emergency. This can lead to misunderstandings about what AI can and cannot do. Serious proposals are then made to simply let AI control nuclear power plants due to a lack of available specialist personnel. We’d rather not do that.
The potential technical protection of AI systems is probably more advanced than protection against such ideas. What is the current state of technical protection for AI systems? Here are a few answers:
- AI systems are just software and hardware. Classic IT security architecture remains relevant, even for AI. On the one hand, this is somewhat alarming, but on the other hand, it means we are at least well equipped to test security.
- There are initial AI-specific protection mechanisms that can at least mitigate simple things like prompt injection. Content filtering and moderation systems can protect against toxic or unwanted output.
- AI systems can be monitored using a combination of statistical and rule-based checks.
- Smaller models such as Small Language Models (SLMs) allow us to reduce the attack surface. Large models such as ChatGPT, Claude, or Gemini are powerful, but particularly difficult to control and test. They are also very large, practically impossible to transport, extremely energy-intensive, and very expensive to maintain. However, there are increasingly better and smarter solutions available.
The more specifically I can define a task, the less I need a general-purpose LLM – and the better an SLM can be used: SLMs are easier to oversee, more transparent to operate, can be hardened locally, and secured more efficiently. These are not panaceas, but important building blocks for responsible AI use. One might ask: If this AI can do so much, why can’t it simply protect itself? Why don’t we build security AI for AI?
Why AI Cannot Simply Secure Itself
As early as 1937, Alan Turing published “On Computable Numbers”, a mathematical description of what he called a universal machine, an abstraction that can, in principle, solve any mathematical problem presented to it in symbolic form. However, the decision problem revealed the limits of machine thinking right from the start. Turing proved that there is no general method for completely predicting the behavior of arbitrary programs. This also applies equally, if not more, to today’s AI.
In any sufficiently powerful formal system, there are true statements that cannot be proven. This brings us to Gödel and his incompleteness theorem. AI can and will become increasingly powerful, even if it will never be completely predictable or understandable. Of course, this does not prevent us from using AI systems.
However, superintelligence will not exist in the foreseeable future. We cannot build AI that is guaranteed to be error-free, and AI cannot do so either. It is interesting and sometimes fascinating, but it is neither a panacea nor a mystery. Our task, therefore, is not to eliminate risks but to identify them, limit them, and bear them responsibly. Pragmatism is called for: we must seize the opportunities while managing the risks.
Optimists say: We can do it.
Pessimists say: It will be a disaster.
Pragmatists say: We have to get through it.
Elmar Geese has many years of experience in IT and IT security. He has been a member of the Greenbone management team since 2018 and a member of the Greenbone AG Executive Board since 2023.
He is particularly interested in the topics of management, security and the management of security, so-called artificial and human intelligence, especially in the context of cybersecurity.
As a trained musician, he still enjoys playing various instruments and is a great fan of classical music.
