Musings on Agentic AI security

I was chatting with a former colleague of mine, Michael Wasielewski over at Generative Security, about GenAI security. He has some interesting ideas around ways to secure the usage of Gen AI but we also got chatting about how we can apply some general security approaches and principles to agentic AI, in particular things like Zero Trust and Secure by Design. There are some good, accessible, overviews of Agentic AI from IBM and nVidia that provide a bit of context for the rest of this post. I’m putting this post out on my personal blog because these are very much personal musings rather than anything that I’ve bounced around with my more learned colleagues at my employer, any blunders in the below are purely my own.  (This blog is called Security ¦ Life ¦ Musings for a reason – and this post is very much of the musings genre 😊).

When thinking about how to apply Secure by Design and Zero Trust principles, e.g. Assume Breach, to agentic AI, it’s important to understand the architecture that we are trying to secure. Consider the tooling architecture shown below (courtesy of Agentic Architectures: Securing the Future of AI with Zero Trust – Generative Security):

We need to consider a number of different architecture layers and levels, and I don’t just mean the Planning, Tool and Validation agents shown above. Each of those agents will have inputs, instructions and context (system prompts), and outputs. If thinking about your threat assessment as part of Secure by Design, then you need to think about who can attack each agent, at each level, and the attack vectors available to them. Another key element to consider here is Time of Check vs Time of Use. If you only do your security checks once, at the entry into the pipeline, what assurance do you have that the agents cannot be manipulated into producing malicious prompts further down the chain? What happens if your agents understand encoding differently or apply different filters to the prompt than that used in your “front door”? Can a bad guy influence the prompts generated within the agentic flows? Can a bad guy influence the flows defined in the Planning Actor? Perhaps even re-directing to agents or RAG tooling under their control (Agent-in-the-middle anyone?)?

“Assume breach” should apply to all agents, the initial prompt AND any prompts generated within the flow. This does not necessarily mean that you need to proxy each and every prompt within a flow - this would come with a heavy performance overhead - but that you should be aware of this risk and manage it appropriately. One potential way of addressing this may be to allocate trust weightings to your agents, based on how much effort has been put into securing those agents and the assurance of those security efforts. Perhaps some form of “taint” could be applied to prompts that are received from those agents with a low assurance weighting? I’d suggest that this taint approach could also be useful if dealing with agents known to suffer from high levels of hallucination. Perhaps another factor could be applied to prompts received from agents known to suffer higher than normal rates of hallucination? Your trust in the final output would then be weighted by these two assurance and hallucination taint factors. This approach does open up another interesting consideration though, as to whether the concept of factors relating to security assurance and rate of hallucination places limits on the lengths of the agentic tool-chains that are capable of providing trusted, reliable, outputs. As with any resilience/availability style calculation, if your key factors indicate less than 100% reliability, you can rapidly get to a place where the end result is less than ideal when you chain the components together. (Of course, use of weightings/factors like these are indicative of the wider move away from binary trusted/untrusted thinking towards more context-based security decision-making based on thresholds - back towards that zero trust way of thinking).

What about the agents themselves? Do you know what system prompts are applied to provide some guardrails to the user prompts they receive and process, or who is able to amend those system prompts? Are those prompts within your control or are you using a service/agent/model provided by a third party? How much trust do you have in that third party? How much visibility do you have of the change control approach to those system prompts? In line with the theme of this post, we should apply the “assume breach” principle more widely – including to any third party agents, their vendors and their users (e.g. if the backend model trained on user prompts then there’s some model poisoning risk).  From a more traditional security perspective, we also need to consider how the agents authenticate to each other and the authorisation controls needed to provide the guardrails you might expect around approved usage of each agent. If we are talking about authentication and authorisation, then if follows that we also need to consider what we do around agent identity and entitlement management.

And then we get to the outputs. Aggregation has long been a tricky issue in information security – bringing together bits of information that are, by themselves, perfectly benign but which, when brought together, may suddenly become very sensitive. In the UK HMG context, this could be where you have lots of individual OFFICIAL data items which when brought together represent SECRET or above through either inference (i.e. if fact a and fact b are true, then so is fact c! Where facts a and b are both OFFICIAL, but the combination of the two (fact c) is significantly more highly classified) or simply through the number of items – think the impact of losing one tax record vs the impact of losing tax records of the nation. So, one consideration would be whether the agents are pulling together information that is significantly more sensitive in aggregate than the component parts. Another consideration here may include whether an attacker has been able to misuse the agents to generate output that is not relevant to the designed purpose of the system – resource misuse.  A further consideration may be whether an attacker is able to use the system to direct a user to a malicious destination, perhaps by tricking a part of the system into recommending a visit to a malicious URL under their control. The Validation Actor in Michael’s diagram above has some important work to do!

So far, I’ve been taking a more architecture-based approach to discussing these issues, looking at the components within the diagram above.  If you read through the articles that I linked to at the start of this post, you’ll have seen that the likes of IBM and nVidia also talk about Agentic AI through the lens of process, e.g. the nVidia post says:

“Agentic AI uses a four-step process for problem-solving:

1. Perceive: AI agents gather and process data from various sources, such as sensors, databases and digital interfaces. This involves extracting meaningful features, recognizing objects or identifying relevant entities in the environment.
2. Reason: A large language model acts as the orchestrator, or reasoning engine, that understands tasks, generates solutions and coordinates specialized models for specific functions like content creation, visual processing or recommendation systems. This step uses techniques like retrieval-augmented generation (RAG) to access proprietary data sources and deliver accurate, relevant outputs.
3. Act: By integrating with external tools and software via application programming interfaces, agentic AI can quickly execute tasks based on the plans it has formulated. Guardrails can be built into AI agents to help ensure they execute tasks correctly. For example, a customer service AI agent may be able to process claims up to a certain amount, while claims above the amount would have to be approved by a human.
4. Learn: Agentic AI continuously improves through a feedback loop, or
   “data flywheel,” where the data generated from its interactions is fed into the system to enhance models. This ability to adapt and become more effective over time offers businesses a powerful tool for driving better decision-making and operational efficiency.”

Now, this post is already a little longer than I was intending when I began it, and so I am not going to repeat the task of applying security principles through this process lens – but I hope that it’s clear that you can. But I will pick up on that last point “Learn”, as that is one that isn’t straightforward to map on to the architecture components. Agentic AI has the capability to learn so it improves its performance against the expected objectives – it’ll do this via a reward structure where certain behaviours are encouraged but others are discouraged. Given the nature of this post, you can probably guess where I’m going with this point. Who controls the reward structures? Is it possible for an attacker to game those reward structures so as to lead the agentic AI towards the preferred outcomes of the attacker rather than the owner of the system?

And with that, it’s time to bring this post to a conclusion. I’d like to think that I’ve demonstrated that there is value in applying general security principles such as “assume breach” and Secure by Design thinking to the world of agentic AI, and that securing such systems is unlikely to result in binary decisions that an outcome is trusted or untrusted, secure or insecure, reliable or not. Furthermore, that it is important to get the level of abstraction right when talking about agentic AI systems. It may be tempting to treat such things as a black box, with a set of inputs and a set of outputs and some magic that gets us from one to the other. From a security perspective, I don’t think we can afford to ignore that magic in the middle. Besides, that’s where the interesting and fun problems sit, so why deny ourselves the pleasure?

Some musings on practical delivery of change

I thought I’d do something a little different for my first post in 2025. I’m usually something of a security magpie, writing about some fun technology or new(ish) development, but this time I’m going to write something a bit fluffier, about some lessons learned delivering real change in large enterprises. For context, I’ve been lucky enough to work on some seriously large technology-enabled transformations across both private and public sectors; the things I will write about below are drawn from practical experience. Of course, it’s my own practical experience and so others will have different perspectives and preferred approaches, but I’m hoping enough of it is transferable to be of some value to others!

Firstly, it’s important to understand the context of the change you are hoping to make. Know where you want to get to (and why!) and where you are starting from. This covers a wide range of considerations (not in any particular order):

• Business context – why do your stakeholders want this change to happen? Do you still need to do a job of convincing the stakeholders to fund the change? Change requires cash. What will the business get for its money?
• Stakeholder context – do you know who the key stakeholders are? Who will pay for the change? Who will be impacted by the change? Who will see the change as a positive and who will see the change as a negative (either personally or organisationally)? Don’t assume your stakeholders are idiots. If you see something that looks plain odd or weird, ask around and find out why things are as they are. Yes, sometimes things are horrible but there is often a reason behind that status. (And, yes, sometimes folks do make curious decisions – but validate, don’t assume).
• Technology context – can you actually do what you want to do, within the timeframe you want to do it, with what you know of the current and target technology landscapes?
• Agree, and enforce, the Vision.  I can’t stress this one enough. Everyone needs to know the target state and why it is the target state. The Vision needs to be owned and supported by someone with enough organisational heft to stop different parts of the organisation from diverging from the overall Vision. You may get folks who are not 100% aligned with the rationale, approach or Vision itself – you need a suitable authority to be available to corral dissenters (alongside offering an opportunity for constructive input). Perfect is the enemy of good, better than current state is better than current state. Progress is important and you can’t get dragged down by continual discussion prior to doing stuff – not least because you are burning time and money whilst you wait, plus risking missing out on the benefits of the change (particularly if they are time-critical).
• Agree success criteria and definition of done. How will you demonstrate that the initiative has successfully completed? What metrics will you use to demonstrate progress along the way?

Okay. So let’s say that we now know who our key stakeholders are, what we want to do and why we want to do it. We then need to do it.  Some thoughts…

• Skills. Know when you need specialist support. Don’t try and force generalists into specialist roles. Some degree of stretch is good – it gives folks an opportunity to learn – but don’t put people in roles where they are being set-up to fail. Mentor and support, get them help if needed.
• Trust. If you’ve spent time putting together your team, trust them to get on with delivery. Do NOT micromanage. It destroys enthusiasm and only corrodes team spirit. If you don’t trust someone to do the job you’ve asked them to do, bring in someone you do trust. The folks on the ground will have a better handle on what is happening from a practical delivery perspective than programme leadership. Listen to their concerns and help to remove blockers. Advice from leadership can be helpful. Cross-organisational insight from leadership into the context behind the blockers can be useful. However, if delivery folks are escalating an issue, it’s usually because they want some help – giving them further actions is not helping. See what burdens you can take off them and what barriers you can remove for them. Help and lead, don’t hector and belittle.
• Lead by example. Every so often, teams will need to work long hours and/or do tedious work. Roll your sleeves up. Do some of the nasty stuff if you can. If the team sees that you are willing to spend long hours buried in the latest Spreadsheet of Doom, then it helps to build a degree of team spirit – that we really are “all in this together”. If you commit to do something for your team, do it. You expect that of your team, they have every right to expect that of you.
• Track progress. Yes, project management matters. I’m not going to pretend that this is the bit of the job that I most enjoy, but I do recognise that we need to be able to demonstrate progress to stakeholders. (Seeing milestones hit, or backlog items delivered, is also good for team morale. People like to see their efforts having an impact!). Know your critical path and dependencies, work backwards to make sure that what you want to achieve is achievable given wider constraints.
• Communicate. This ties into the above… you spent a lot of time identifying your stakeholder community, you really need to keep in touch with them. Let them know how the initiative is proceeding. Ask them for help if you need it – senior stakeholders are often keen to help as it justifies the time they are spending meeting with you.
• Meet with purpose. This ties into the above… (again). Communication is good. It’s much better if meetings and communications have a clear purpose though. If you’re getting time with senior folks, be very clear about the purpose of the meeting, be diligent about preparation for the meeting and have some defined required outcomes. I’ve found few things irritate very senior folks like having their time wasted through attending a call where the preparation hasn’t been done, resulting in the need for yet another call. Don’t irritate very senior folks without due cause, it’s not helpful.
• Delivery methodologies. Pick the right tools for the job – whether project management, architecture frameworks or industry standards. But don’t be dogmatic and do NOT assume that everyone has the same understanding of what you may think of as standard industry terms. Establish a common taxonomy as part of agreeing the overall methodologies.
• Quality is important. I’m a pedant. I know this irritates my team on occasion. However, I am unrepentant. I think it’s important for documents to be of a high standard, typos and grammatical errors are distracting for the reader, whilst any internal contradictions or lack of appropriate definition of terms just leads to confusion or (worse) incorrect interpretation and follow-on action. Be clear, be concise, be accurate.
• Respect reality. Requirements may change during the course of an initiative. You may encounter unexpected obstacles, perhaps even insurmountable obstacles, during delivery. I’m not going to go into the basics of change management here, but I do want to stress the importance of recognising when things move from difficult to (practically) impossible. Don’t risk burning folks out trying to do the impossible.
• Enjoy yourself. We only have one life. We spend a lot of that life at work. Try to be a nice person to work with, or work for. Respect others, help others, try to have fun – just because what you’re doing may be deadly serious, it doesn’t mean that you can’t try and enjoy the experience.

And I think that's where I’ll stop for now. It’s probably a bit motherhood and apple pie for those who have been around for a while. I acknowledge that there are lots of books out there around project delivery (although I do keep having an urge to write something around enterprise security architecture and change, apologies in advance should I succumb). I’ve just had an itch I wanted to scratch for a while about jotting down some of the things that I’ve found helpful over the years, alongside some of the things that I’ve found less than helpful. I sometimes see folks entering the security profession wondering why things don’t “just work” or don’t just get fixed. I’m hoping the above helps to provide a bit more insight into the kind of things that need to happen, or at least should happen, to actually drive positive security transformation in large organisations.

 

Itch scratched.

 

Stereotypes, context and situational awareness.

One of the things most likely to put me into a strop is being told that "you can't do that!", particularly when such a statement is accompanied by absolutely no rationale as to why not. One common example of this is whenever I am told, or hear, or read, blanket statements about what CISOs (or other senior leaders) are, or are not, interested in.  "You can't say that to a CISO, they're just not interested". Oh really?

I have a few issues with such a blanket statement. Firstly, lived experience. I'm not a great fan of being told that some of the things I've done didn't happen. Secondly though, don't you think it's a bit simplistic to stereotype all CISOs as having the same interests, with the organisations that they work for all being in the same position with the same dynamics?  Consider the simple chart below:

You probably already know where I'm going with this. You can plot both individuals and organisations against these axes. There's also a time dimension - for example, a CISO may need to come in to fix a broken security function before then settling into a steady state. Individuals will often be more comfortable in specific quadrants - some folks love the challenge of driving difficult change, others love maintaining order.  There are no value judgments here. Likewise organisations: some will be in a good state and looking to keep things ticking over as they are, others will be in the middle of the churn of a fundamental transformation. It helps when you have a match between the leaders and the organisation! But that's a different topic. Anyway. Depending upon where the CISO is sitting at the time of your conversation then you may well find that they are interested in different topics. Clearly, going to a meeting with an assurance-focussed CISO, comfortable with the status quo in a stable and mature organisation, and trying to talk about the practicalities of moving towards zero trust, security by design or devsecops is unlikely to go well. Mention of DAST, SAST and SBOMs may well induce a few eye rolls. However, if you're talking to a CISO that has staked their personal reputation on delivering just such a transformation, don't you think they may have at least a little interest in how they could protect that reputation by talking through the "how" of how such a transformation can be achieved in the real-world? I'm not suggesting we have to talk 0000s and 11111s, raw TCP/IP or any other deep technical jargon relating to security transformation (although some modern CISOs have grown-up in the industry and it's not necessarily an alien tongue to them), but we certainly don't have to limit ourselves to platitudes and quotes by our favourite analysts.  Yes, there are some commonalities in the role of a CISO; the need to be able to manage upwards, the need to be able communicate with (and influence) business stakeholders, the need to be able to manage budgets, the need to be able to nurture and shield your team etc. However, if you find yourself being told that "the CISO won't be interested in that", then try asking the person telling you that whereabouts on the chart they'd place the individual CISO they have in mind. They may be right. But they may not be. It's probably worth finding out.

Zero Trust - a little light grumbling.

I think I've reached the conclusion that if I haven't annoyed at least 5% (arbitrary figure) of my audience when talking about Zero Trust then I haven't done my job properly. Too many competing definitions, too many strongly held sacred beliefs. Naturally the next step is to see how many folks I can annoy on the Internet :). So, definitions for starters - I use NIST SP800-207 and the CISA ZT Maturity Model (now at v2) as my baseline. Vendor-agnostic. Analyst-agnostic. Wide in scope. Great fun for annoying folks who insist that ZT is purely focused on Identity or the Network*.  I do then try to simplify the topic:

⁌ every access request starts from a position of zero trust (applies to all entities - humans, devices, services)
⁌ authorisation is granted based on dynamic context (risk-based)**, ideally per request
⁌ assume breach - of user ID (including machine or application service ID), access device, transport network.

What does this give you? Well, you've done away with the arbitrary distinction between "inside" and "outside". You now need to do something about those legacy flat networks. Reduce your blast radius! You can also now give your users access to the business applications they need, wherever they (or those business applications) may be located.

You've also now got to do something about your machine (OT, IoT) and workload (VMs, cloud instances, containers, applications) entities so that a compromise of such entities doesn't mean easy traversal.

You're assuming breach, this means you should be embedding observability into your in-house developed apps and configuring everything else to generate the signals you need to automate and orchestrate those dynamic authorisation decisions.

In short, improved access for legitimate users, dynamic per request authorisation based on current context (including risk) for all entities and better visibility across your IT ecosystem, enabling faster detection and response.

I'm not sure why delivering those security outcomes remains a little controversial. Perhaps Zero Trust is just another one of those labels (like cloud) that rubs folks up the wrong way. Look past the label. Oh, and don't get too attached to specific definitions. Except the ones I like. Those ones are fine. 😇

*both key pillars to be sure, but not the sole focus.
**dynamic context - which is why your network microsegmentation is not really Zero Trust.

Growing pains?

An interesting story here on the legal status of relying upon US-owned cloud services for the processing of law enforcement data - https://www.computerweekly.com/news/365534023/Scottish-police-tech-piloted-despite-major-data-protection-issues.  The conflict of such processing with the obligations stated within Part 3 of the UK Data Protection Act 2018 is something that Owen has been raising for a long time now [Disclaimer: I’ve known Owen for years – he knows his onions] and it is good to see these issues now being explored more widely.

For me though, this is part of a wider re-evaluation of the usage of the cloud hyperscalers. Consider also the context of financial services regulators across the globe expressing increasing concern about systemic risk and how the reliance on a small number of hyperscale cloud providers impacts upon the current push to improve operational resilience across that sector.  Speaking of resilience, how comfortable should we be with quite so much of our public sector and other providers of Critical National Infrastructure (CNI) services being fulfilled by the same limited pool of US-owned services?  There is increasing discussion of Sovereign Cloud approaches (e.g. https://www.capgemini.com/insights/research-library/cloud-sovereignty/) however, in reality, can such sovereign solutions compete with the hyperscalers? The experience of UKCloud, an early entrant into the UK cloud market suggests it is a rough ride for smaller players (they were placed into compulsory liquidation in October last year).  Should pure commercial considerations be put aside and Government subsidies made available to provide safe, legal sovereign cloud services?  Can any of the hyperscalers derive ownership structures that provide genuine confidence that their “sovereign” solutions offer sufficient protection from US over-reach via the Cloud Act?

So, am I saying that we should avoid the hyperscalers? As ever, it’s more complicated than that (is that framing taking over from “it depends” as the consultant’s phrasing of choice?). The advantages of cloud services remain – for many the infrastructure, security and physical hosting services offered by the likes of AWS, Azure and GCP surpass those available using existing technologies and skillsets.  They have greater budgets for innovation, greater elasticity and, these days, a growing pool of certified talent able to deliver value to cloud consumers – at pace. I do however think that there is a growing conflict between the needs of individual organisations and the needs of wider sectors, their regulators and wider society.  The former (quite rightly!) want the best bang for their buck whilst the latter are more worried about the “severe, but plausible” events that may lead to catastrophic consequences.  I remain a big fan of the capabilities that the likes of Microsoft, Amazon and Google offer their consumers.  I remain of the view that, in the majority of cases, a new, well-configured, cloud-native solution will likely be more secure than a solution delivered through legacy alternatives. But there are tensions.  Looks like we are approaching the point where those tensions need to be properly explored and informed actions taken by both regulatory authorities and governments to better balance risk and reward for the societal needs that they are there to protect and serve.  Thoughts?

Security hysteria - time and a place...

...which is not now.

Originally posted this on Facebook but it's probably more of a blog post!

I see a few people worrying about cybersecurity at the moment due to increased use of tools like Zoom. Let me give you a bit of context.

When a service suddenly becomes more popular (hello Zoom), it draws the attention of the security research community. Bug hunters find bugs. If having security bugs means you don't want to use a tool, I've got some very bad news about the other apps on your PC/Phone that haven't yet been subjected to (public) scrutiny...

Is Zoom "safe"? It's likely as privacy-safe as other Internet services - I'm typing this on Facebook for God's sake. Are you a national government? Are you a big bank? International criminal? No? Then if you find it useful you should make a decision based on whether you believe employees at Zoom will have any interest in eavesdropping on your friend group... but adopt usual Internet good behaviours: don't click on links unless you are confident they are safe, only download any client software from official sites, put passwords on your meetings so you don't get bombed, apply patches as they are released.  Am I guaranteeing that Zoom HQ will not get hacked? That the Zoom client will never get backdoored? Of course not. But at this point in time, I'd suggest seeing some friendly faces is perhaps worth a little risk?