Defining Trustworthy AI Before AI Scales Beyond Control

Hi, I'm Joan Goodchild with CSO. We are here this week at the CSO Cybersecurity Conference & Awards, and I'm thrilled to be joined by two guests who are also attending the event.

Malcolm Harkins, he is a veteran CISO and the currently the Chief Security and Trust Officer with HiddenLayer, as well as Gary McAlum, he's a Senior Advisor with Oliver Wyman. So, thank you both for joining. Thank you.

So of course, big central theme this week is we're talking about AI, everything AI and so many security practitioners are still really kind of struggling with the concept of, you know, how do I keep up with the innovation of AI while also securing it.

So let's kind of start right there. Malcolm, let me start with you. What does trustworthy AI mean?

Yeah, you know, it's a great question. So I think in the way in which the industry has been framing it for probably five, six, seven years has been more around that the AI doesn't have a bias and it's responsible and it's fair and all that stuff.

I think that is an inaccurate way to describe what trustworthy AI should be. I think that trustworthiness scale of its ethical, it's legal, it's accurate, while useful, is missing out the core security components that provide the assurance of the trustworthiness.

I need to have the supply chain security, but my security development lifecycle. Is the model good or has it been tampered with? I need to have kind of basic guardrails and runtime protection to negate simple things that could impact the model.

But to truly get to trustworthy AI, I think you need to have kind of the advanced behavioral, kind of real time monitoring of the inputs and outputs and the adversarial aspects and the operational aspects around protecting the models.

Because if you don't have that, I can guarantee you won't have trustworthy AI.

But the way in which the industry has framed it and including most of the existing AI related regulatory things is around kind of those legal and ethical aspects and it's missing the real operational security components.

Yeah, yeah, I think that's exactly right. I think that there's sort of a textbook definition. If you look at the NIST AI Risk Management Framework, there's like eight to 10 characteristics of trustworthy AI.

If you go to the EU AI act, there's references to, you know, governance and trustworthy AI, ethical AI type of considerations. I absolutely agree with Malcolm on that.

There's an operational context that I think adds the richness to it in one of my former roles as Chief Security Officer, I own the fraud team, for example. And I would say from a practical example, the fraud team depended on fraud detection models.

In today's world of AI, if a fraud detection LLM performs as expected, despite adversarial attacks or attempt at manipulation, then that's trustworthy AI. So it's got to be able to do more than just be validated, be secure.

It's got to be exposed to rigorous testing and validation and adversarial mindset.

So let me put you on the spot a little bit then, based on all that, what is required then to actually achieve trustworthy AI with all of those kind of factors around modeling and so forth in mind?

Yeah, I would say it takes a disciplined engineering approach that looks at the control environment from an operational reality that may not exist beyond the existing frameworks and guidance that's out there today. So very disciplined, very operational.

Yes, you have to have controls and all those things, but you have to look at it in a very different light than maybe previously. You did look at something. Yeah, I'll share.

Yesterday, when I was giving a keynote, I shared a case study on something that one of our researchers had uncovered late last year. We had an employee went to a doctor's appointment, physical exam, Doctor says, hey, I've got this physician's assistant type thing.

It's going to listen, help do these things. And the employee's like, yeah, you can use it, but I want to know the name of the solution. Doctor gave it to him. Employee gets back to work. Slacks are researchers within under two hours. We owned everything.

This physician's assistant capability is used in over 50 countries, processes over 2 million patient records a week. It had SoC2, it had ISO, it had all the privacy compliance, it had everything. It was, quote, unquote, trustworthy. And guess what? In under two hours, we owned it.

Because they were shipped for brands.

When it came to actual operational security of the model, and once were able to get to that, were able to get to their OpenAI keys, their anthropic keys, AWS, Google Cloud Stripe payment system, were able to access and touch every connected health care capability.

Epic, all those things, we could have taken out the entire health care system. We could have dumped every health care record, we could have modified things. Why? Because it was lacking the real runtime monitoring security for adversarial attacks against the model and against the agent.

Yet it was, quote, unquote, trustworthy.

Yeah, I mean, the implications are obviously pretty breathtaking for what is possible. And again, because it's such a central theme this week, you know, I've been talking to a lot of different people about, you know, just kind of the implications.

And one comment that really stayed with me was it's. It's one place where innovation has kind of arrived long before security. Like, we can't even consider how to secure this technology because we're still, you know, sort of trying to figure out how we're using it, what we're doing.

I mean, you said before we started here that you're talking to a lot of people about just how they're struggling, you know, with exactly what direction to go in. Can you speak to that a little bit? Yeah.

I think this is an overabundance of opportunity here. Right. That's creating this sense of what do I do first. Right. I mean, there's so many options here. And I think there's this great saying that I used many years ago, right? Do something, but don't do nothing. Right.

And I think right now you've got to understand your environment, first of all, right? You've got existing technology stacks, you've got processes in place. You can't take the old processes and just do them faster. You're not going to patch your way out of this problem here.

So you have to look at things very differently. And I think the prevent, detect, respond model of yesteryear, it's still a valid model, but you're not going to do much prevention in this space.

So you're going to have to get much better at detecting and responding and where you can be proactive in terms of employing technologies that can help deal with this increased velocity of the bad guys. I think that's where you're going to start to make some progress.

I loved one of the things that Malcolm talked about in his talk is he talked about the old risk management equation. Risk equals threat times vulnerability times consequence. I used to add probability in there as well, but he changed the vulnerability factor to exploitability. Right.

And when you recognize, like, everything is bad out there. Right. But not everything gives you the same amount of risk exposure that starts, I think, to help you mature your risk management approach. Right. You can't patch everything. What are the worst things?

And, you know, from a practical perspective, as a ciso, I remember a CVE would come out with a critical severity rating and everybody say, what are you doing to patch that? I said, well, it depends.

Maybe a CVE with a critical vulnerability rating that's embedded in the environment with compensating Controls is less exposing than a medium that's on the external facing perimeter. Right. So we had to put context in a more of a mature approach that I think this is the same thing.

I do think the technology can benefit the defenders and it will. But do something, but don't do nothing.

That's what I'd say now, Malcolm, I.

Mean also with your depth and years of experience too, you must be here having a lot of these conversations as well and hearing this. And we talked about modeling again kind of at the beginning of this conversation.

But you know, what are your thoughts in terms of how existing models kind of address the new concerns around AI models or agentic AI security?

Yeah, the one thing that's very clear is the existing $270 billion we spend on infosec and all the existing compliance regimes do nothing to protect AI. They protect the infrastructure that the AI is operating in, but they don't protect the models or the agents that from adversarial behavior.

So one, don't assume that your existing controls will do anything other than protect the infrastructure that it's already protecting because it wasn't built to do anything more than that. I think number two, and you made this point of people are still trying to learn and struggle.

Okay, yeah, that's true. That's always been the case of new technology. But the people who frankly whine about that get over it and do your damn job. It's not that you go back to the basics, understand the risk, understand your exposures and take action.

It's a knowing versus doing gap. Now I do believe, and I'll just do a shameless plug for Hidden Layer. Hidden Layer launched as a company seven months before ChatGPT ever did.

Probably one of the first times in history that you had security capabilities that were meant to protect things before the watershed moment. The problem we have is a knowing versus doing gap. And that's the thing that we always need to continue to shrink.

And I've engaged with a lot of peers and stuff like that and some of them are doing and taking the progress. Some of them are taking a wait and watch a seat because they're using risk as a rear view mirror driving.

They're like where is it in the system known exploitability list? Where is it in the news that the AI models are being attacked?

That is a backwards looking view of risk versus what I believe my job is not only to understand that, but again to what Gary said. Look at my exposure. Am I exploitable? Are my models and agents exploitable? And if the Answer is yes.

And it's going to create an oh, shit moment, a material event for your customers or your business. You need to act. And when you look at it that way, I think there's two choices. You disclose it or you deal with it.

If it's a potential material event, those are your only two choices.

I would add to that, I think there's another group of people out there that are being driven a little bit about the compliance mindset. Like, what do we have to do? Right, right.

You know, EU act, if you're in the European side, maybe the EU Air act is going to drive some of that. That's a very much of a governance type of model. Some people will look at the NIST AI Risk Management Framework, which is good.

NIST CSF will also evolve to account for this. But existing frameworks today and regulatory standards don't fully take into account the operational realities of an AI agentic environment. They don't. So if you're waiting for that, you are way behind the power curve. That's the other thing I see.

People are waiting to be told what they have to comply with.

I remember years ago in a previous job, as I was briefing a board, I remember they were asking the question, are we compliant? I said, well, that's a different question than are we secure? Right.

And I think that is, if we're still stuck, if you're still stuck in the mindset, we'll do what it takes to be compliant. That won't be sufficient. Right, right.

And I mean, we've been having this conversation now, you know, long before were talking about AI, you know, for 20 years, that compliance does not equal security.

So, you know, is there, I guess, Gary, you know, as a follow up to that, like, is there any distinction or difference here with the concerns around compliance and AI, or is it really still sort of the same formula, only just a different technology opinion?

I think it's the same type of formula. Right. There are things where you can be compliant. Right. And you can be compliant with, you know, the EU AI Act. Right.

Have all the governance and reporting in place, but that's not going to tell you how to secure your agentic AI models. Right. It could tell you what you got to do to be compliant, but you still got an operational context.

It's not going to tell you how to do that. So that's the dichotomy.

Yeah, I agree with Gary. And even if you look at the ISO 42001, which is meant around AI, again, it lacks specificity for the operational security and the monitoring and the detection and response.

And again, you go back to 20ish years ago for us, you know, as Gray, you know, we saw the explosion of Dwell time in our environment. Why? Because we didn't have the refined advanced monitoring in our infrastructure? Well, we by and large, reasonably solved that problem.

And we're leaving the agents and the models unprotected. And guess what? That Dwell time, I can guarantee you, is exploding and it will bite people in the ass. It might not be in the news right now, but it will occur.

All right, well, that is our time, but it's been a great conversation. It'll be interesting to see how this unfolds. Of course, always interesting in security, right? Gary McAlum, Malcolm Harkins, thank you very much for joining us today. Thank you.

Thanks, thanks for having us.

And thanks to you for watching. For now, I'm Joan Goodchild with CSO. We'll see you next time.