All of the posts in this blog are solely my personal opinion rather than those of my employer. This one is particularly Lee-centric, i.e. my thoughts, not those of my employer.
There's a lot of noise and heat at the moment around AI and cyber. There's more heat around the anthropomorphising of LLMs and agents, including in the cyber domain. For the purposes of future reference, I just want to capture a few of the assumptions that are driving my current thinking when it comes to the evolution of AI and cyber.
1. Agents are not human. They are however goal-driven and trained to recognise good and bad behaviours (alignment). Agents will sometimes tackle problems in ways that humans would not.
2. Alignment is not the answer for shepherding agent behaviour. Humanity does not have an agreed set of what is good and what is bad. These days, even the Golden Rule appears to be somewhat less golden to some. If we can't agree on some basic rules, we don't have a foundation on which to train our models. The ethical stances and requirements of model producers vary.
3. LLMs are sensitive to initial conditions. They are non-deterministic in that we can ask the same question twice and get different answers.
4. Lots of independent entities with a sensitivity to initial conditions seems a situation ripe for some interesting emergent behaviours. We see this already, e.g. [2608.26081] SwarmWorld: Stigmergic technological evolution in societies of language-model agents.
5. Agents are good at security research. Don't believe me? Look at the number of findings addressed in recent updates from folks like Microsoft, compare to the old baselines. The impact is real.
6. Not every producer of software has the resources of a Microsoft. Some issues will not get fixed in a timely manner.
7. Established security patterns remain valid: Zero Trust. Least Privilege. Microsegmentation. Patching. Observability. Attack surface reduction. Add in deception technologies and most large organisations should be relatively safe from attack by agents from the outside.
8. Most organisations do not have all of the elements listed above in place.
9. Organisations also need to protect themselves from their own agents. Whilst blanket technologies such as those listed in (7) above are helpful, technologies around agency and intent are still evolving - the blanket technologies provide the hard guardrails, but softer guardrails are also necessary to enable business value.
10. The open weight models are circa 6 months behind the frontier labs.
11. The frontier labs have proven unable to secure their agents. It seems likely that Joe and Josephine Bloggs may well not know how to even try (or even that they should try). Open weight capabilities without sand-boxing or alignment will eventually proliferate.
12. The sky is not falling *today*. This provides no guarantee for *tomorrow*.
My overall take: the impact of AI on the security environment will largely be determined by the race to implement good security practices vs proliferation of AI capabilities. If we look at how long typical security improvement programmes take to implement, I remain nervous about the resilience of our Internet-based society should we have hundreds of thousands of capable agents attempting to solve problems however they see fit before our defences are ready. (i.e. agents do not need to have malicious intent to cause problems, they just need to explore areas of the problem space that humans would not).
Addendum:
When we talk about AI and security, we probably also need to be clear about the frame of reference we are using, as there are multiple perspectives available, e.g.:
- Model producers
- Organisations
- Organisations needing to secure their own AI implementations
- Organisations needing to protect themselves from the AI implementations of others
- Individuals
- Governments
- Legal frameworks (e.g. who has accountability for agent actions?)
- Regulatory frameworks (e.g. how should agents be used?)
- Societal impacts of AI (e.g. job losses, lack of entry roles)
No comments:
Post a Comment