Fireside Chat: Incorporating AI into Security at Adobe

Why WAF Rules?

[00:00] Jenn Gile: So, you know, you used all of these AI/ML techniques to improve the way that you're able to write WAF rules, both faster and perhaps even better. What made you and your team specifically decide WAF rules is the right place to start when using AI for security engineering?

[00:23] Ammar Alim: Yeah, that's a good question. I'll also give a background about how this topic came about. It's actually came about on a recent interview that we were conducting. So we typically like to provide candidates with a take-home exercise project related to what we do day-to-day. We feel like abstracted interview questions may not be relevant to the day-to-day. So, 'go do something, let's think about something that we actually need inside and see what we can if you can do something about it.' Just a problem-oriented interview process.

[00:59] Ammar Alim: What we why we decided to do this is one, we have six WAF vendors, and we have a team of the WAF team is about three people. We have six vendors and we have about 90 product teams, 90 products at Adobe. So you have the Photoshop Web, the Illustrator, Adobe Experience Platform, you know, the list goes and on and on. We cannot be servicing those teams using three people with the quality that we want. And LLM came out and really disrupted software engineering and I think the security area is really ripe to be disrupted as well, just due to the availability of data. LLM thrive when there is a lot of examples, and there's a lot of HTTP examples out there as far as traffic goes. Me and you sitting here, there is gazillions of requests going around the world, people trying to access services, use mobile or computers, even, you know, students at schools are using their laptops to access online testing and so on. So there's so much examples and the the more examples there is, you can notice the difference as far as the LLM performs.

[02:00] Ammar Alim: And we're not just using LLM. We know LLMs has limitations, that they can hallucinate. We, you know, during the interview this came up, like, how could we mitigate the shortcomings of LLM? And we said maybe a RAG. A RAG is something that, you know, it's a database that enhances your LLM with recent, most up-to-date knowledge. So I can build a knowledge base and feed it to the RAG, and the LLM when retrieving and generating content for for us, in this case, is a WAF rule, they can look up the most up-to-date knowledge base. We could say for Photoshop do not create bot rules, but for AVP, go ahead and do that. Um, you can also have an MCP server sitting on the side with the most up-to-date vendor documentation. So, the LLM get assisted to make the right decisions and the correct decisions, and we still review things. Like, the danger here with the AI, we feel like not just AI would WAF generally, we need to be able to make sure the rule does not block legitimate traffic. And we have a system for that. We roll it in a staging, we validate that is working, before we say, "Okay, this is clean and it can be moved to production." So, in short, I know like this is a long answer to to a question, but in short, we we cannot scale the team using like classic detections which is detection as rule, similar to the industry, but now we're entering a phase where we I call it models, a detection as models. So we used to write manual rules to detect things. Then we move to detection as code, which is writing maybe a Python script that um, if conditionally, uh, you know, can be conditioned to write things according to some conditions. So it's more expressive, it's more capable than just one rule trying to detect. Now, we can write an entire model that looks at the data continuously and make some good decisions and you can always make it better and, you know, iterate over it continuously.

AI and Security Engineering Fear

[04:47] Jenn Gile: So I that gives me a couple of follow-up questions to ask. One that I would focus on first is whether you're seeing much fear coming from security engineers about um AI in security engineering specifically potentially, um, if not like eliminating their jobs, um, meaning there will be fewer roles available if people can be more effective. Um, you know, I know you and I have talked about this before, but maybe more on the fear side. Like what are you seeing from security engineers and their outlook of using AI in their jobs?

[05:32] Ammar Alim: I think the, you know, those the fear is there. Some people don't have fear, some people do share concerns, especially I get this concerns from people trying to get in from students that I've mentored. I went to University of Washington recently and I spoke there, and this this question came up. But I the way I see it is AI is enabled developers to create more software. More software means more security problems. More security problems means more human ingenuity needed to solution using AI. So, I feel like people are being said that for a while. If you use an AI, uh, you will be able to, you know, continuously be, uh, useful. If you don't adapt and still trying to keep up with developers using AI, like, if a developer is writing a Ferrari, you cannot catch him using a Mazda. Like I'm sorry, like I had a Mazda so, you know, I'm being um, So, yeah, I think security engineering need to keep up, and to keep up, you need to adapt to the tools that the industry, the rest of the industry, is using to be able to catch up.

[06:50] Jenn Gile: I drive a Mazda also and as much as I love it, you're right. Uh, it's not gonna compete with the Ferrari as much as we would love it to. Um, the second kind of related question I have is more like, um, what will people be doing if they're not writing WAF rules anymore? And, you know, you can kind of draw a parallel with application security engineers if they're not spending a lot of time triaging potential false positives and generating tickets, if you have a tool to do that, you know, they might be threat modeling, they may be working on secure design review. What do you see your WAF team working on once you get this into production if they're not having to constantly write new rules?

The Role of Security Engineers in an AI-Driven World

[07:44] Ammar Alim: So, this system will never stop needing to be trained. Like, this is just given. We have so many different applications, they all process different type of traffic, and the traffic is changing daily. We front a service with some MCP that slightly adjust how traffic look like. We introduce some agents in front of Photoshop that slightly modify how Photoshop behaves. So this system continuously needing someone to be the security expert that tell it what to do. So the LLM is not going to be a security expert. It does not have business context. LLM does not know what Adobe Photoshop landscape look like, right? Like it just have some data scraped off a Reddit, you know, that's it. We need to bring a lot of security experience. So looking look at it as you have this very, uh, capable system that if you teach it well, it will serve you well. If you abandon that and, yous know, hope for the best that it has what it takes to do the job, it's gonna be, you know, you're gonna be disappointed. And you can easily test it. Like try to use just pure LLM without any, uh, a custom model that you train, like in-house, to do this work, and it's just going to block legitimate traffic, it's just a lot of false positives. We really suffered from the like the static code analysis things and we really, you know, moving to more dynamic, uh, universe where I think we should take advantage and I like I said, I think the security space has we have a lot of data, and security is a false-positive problem. So when you have a lot of data, you have a lot and machine learning, you have a a good opportunity to leverage that data to do something interesting. Um, so I feel like security engineers will now be training machine learning systems and building and assembling machine learning systems to complement like vendor tools, to, you know, bring about solutions that previously were not, um, possible.

[10:11] Jenn Gile: Yeah, I mean to your earlier point, if you have six tools supporting hundreds of projects, um, being an expert in all six tools is unlikely. Uh, getting parody between those tools is unlikely. Um, so it, you know, is not just about efficiency, right?

Measuring Efficiency and Scalability

[10:29] Jenn Gile: So, speaking of efficiency, like if we look at the studies that have come out about developer efficiency when using AI tools, it varies from study to study, but generally like when used well, when used right, there is a an efficiency productivity boost. Um, you know, what are you kind of expecting to see once you roll this out into production when it comes to the efficiency of your team? How do you think you'll, you know, measure that it's working?

[10:59] Ammar Alim: Yeah, right now we're literally being paralyzed by the amount of products, the amount of vendors, the amount of volume of logs, right? It's three people cannot just go and support 90 systems. Um, so this project is actually built for developers to self-service. So I can't be in each product team conversation every day, but the best way to use this these custom models is to train a model on a product specific traffic. So we're not training it on an aggregate of Adobe traffic. Each product have unique, uh, problems, have unique, um, type of traffic and users, right? So training on specific use cases give us better signal and this system is not very expensive to run. Um, like I will demo this, it should can run on my laptop. So it's a lightweight model that it can, it can get bigger depending on the needs, but you can start small and explore. Um, so instead of us monthly do doing a WAF review for a given team, the team can daily use the system. So we can move from month or a quarter to daily cadence, weekly cadence. So that's the productivity gain that not just my team is going to get, but each product team at Adobe is going to have a WAF engineer at their disposal that doesn't get tired, doesn't sleep, doesn't need food, doesn't make a lot of mistakes if we train it well. Humans tend to they need to go to sleep. They can be up 24/7. Um, and I don't know about others, but I do not stand Regex. I can't just that it's just not intuitive for me to do it, but AI doesn't mind that. Like AI is very capable as like generating the next predicting the next token needed to solve a problem, and WAF rules with AI are and ML are a matter of I can create 10 rules in a few minutes. That is not possible with a human being. Like it's just not possible.

Addressing Security Concerns with AI

[13:46] Jenn Gile: So, um, I would anticipate your leadership as much like a lot of other companies where there's an expectation that everybody is using AI to be more effective, um, to, you know, drive better scale, better outcomes, but also there's certainly a lot of people concerned about the security ramifications of using AI. Excuse me. And so, as you're, like, you know, you said you're going to be presenting this at a leadership offsite soon. Um, how are you going to talk with them about the ways in which you addressed security concerns with this as you built it?

[14:31] Ammar Alim: Yeah, we treat AI like any other system. Secure defaults, least, uh, privilege, human, uh, in the loop validation, and, um, we plan to self to self-host this internally, and like on AWS, maybe Bedrock, um, or something else. And the same principle apply, like cloud security is still relevant here, right? Um, MCP has been the concern lately because remote code execution, but things has gotten better, and one of the things like we now planning to leverage remote MCPs. We're gonna self, uh, build our own MCP servers and self-host all of these things internally, authentication, encryption, uh, just the basic. Like security, once you get the foundations right, um, and this is where most of us kind of neglect the foundations, and you're never gonna see like a sophisticated attack, um, like it's not someone, um, I don't know, figuring out a decryption algorithm who get access to your data. Is that me, you forgot to do authentication right, you forgot to encrypt your S3 buckets, simple stuff. So, once we handle those, um, we're pretty confident about making sure those systems are secure.

[16:04] Jenn Gile: Yeah, you're making me think of the uh, current wave of NPM malware attacks, uh, most of those are not sophisticated attacks. They're not doing super fancy things. You know, they're just relying on the lack of fundamental security controls like MFA. Okay.

Future Use Cases for AI

[16:26] Jenn Gile: What are some other use cases that you and your team are exploring now that you've kind of proven out that this WAF one has legs, you're getting ready to roll it into production. What's next?

[16:40] Ammar Alim: Yeah, it's uh, for me it's still around WAF. Um, so I'll give you another example that we're thinking about, um, pretty soon. So the way we on-board on a WAF vendor is usually an API. We make some API calls, we build a self-service, uh, on-boarding form. People fill out information. So we would like to actually offer MCP servers that enhance the on-boarding experience. So anyone can self-service on-board. They can't, but will help you, help guide you to onboard correctly. So you don't have to read documentation. So I have a lot of documentation. We we ask developers to step number one, go read the documentation. Step number two, go and fill out the form, and then some automation is going to happen behind the scene. We would like to blend those together where it's one experience. You don't have to read anything. If you are have you have a question, you just ask the chatbot, "Hey, what do I need to on-board?" The chatbot is going to response and say, "You need an Akamai API key or F5 API key." "Okay. Where do I put the key?" "Maybe in a secure location where I can grab it from." "All right, I'm ready. Can you get it done for me?" And we have an MCP server that has access to the API, so it's an extraction layer where they can use natural language to on-board to a WAF vendor. No needing to build anything, no needing to read anything.

[18:19] Jenn Gile: That's really cool. They're gonna love that.

[18:21] Ammar Alim: Um, what does it look like to build that kind of experience in partnership with developers so that you know what they want from the experience?

[18:27] Ammar Alim: Yeah, we typically roll out an MVP and we find a few candidates who can partner with us on, you know, testing it out and we let them play with it and give us feedback. We get the feedback and that's goes into the next iteration of of the product.

How to Get Started with AI in Security

[18:48] Jenn Gile: And then, uh, for somebody who wants to get started with any of the things you've talked about, you know, building their own MCP server, uh, setting up a RAG, um, we didn't talk specifically about training your own models too much, but, you know, the whole gamut. Um, where where would you tell them to start?

[19:10] Ammar Alim: Yeah, um, so most of my mentees are students. They don't have a lot of like a bigger budget. So what I really recommend and the resources are fused, you know, during my earlier days was just Udemy. Um, Udemy, you go to Udemy, the price of lunch will get you a very solid, uh, course. Uh, one course, uh, one instructor that I would highly recommend is it uh, Donner, so E D D O N N E R. He is the CTO of a New York based startup called Nebula, and he has two courses, one on LLM engineering and the second one on agentic AI engineering. I'm actually taking the agentic one right now, really good. Um, as far as books, it's the popular AI engineering book. Most most of us are reading it right now. Um, I really love some of the courses that Claud had produced, like using cloud code, the MCP, um, documentation and course, they also have an MCP server's, uh, course, it's free. Hugging Face documentation and courses, they also have an MCP servers, uh, course, really love it. Um, so start there, um, go to Udemy, even machine learning, like, just coming up to speed with NumPy, uh, Scikit-learn, those are libraries, Python libraries that if you have some Python foundations, they abstract lots of hard concepts of machine learning where you can write a small function that create a, you know, straight line for regression, linear regression or clustering. Like, it's not very hard to come up to speed with this stuff. If you want to do something very sophisticated and very deep, you're gonna need to like a serious book or serious training, but to get up to speed to get started, it's not that difficult. I like to start from AI and then go to machine learning, like get your hand like, see the magic. Um, see the output, see the outcome, but once you need something local, relevant to you, like AI is not trained for just WAF use cases, so that's why we, the, the decision doing the interview to go with something that we built custom for each product team, um, you need ML at some point. Like, you need to be able to do that, but AI can also help guide you to do that. So it's a little bit of training. You can still leverage AI to divide code, maybe, um, a machine learning model, that may be a thing. But I I still feel like the foundations are really important, um, to to have there and Udemy also have a lot of good machine learning, uh, courses. $11, you have a good machine learning, uh, course.

Final Thoughts and Job Opportunities

[22:59] Jenn Gile: I just pulled those up on my machine and they look really interesting. Um, I may be misremembering, but I think you posted on LinkedIn maybe within the last month about, um, an approach to, you know, you've got a book on topic X. How do you do you just read that book cover to cover or do you, um, you know, cherry pick? Am I remembering right that you've written about that?

[23:25] Ammar Alim: Yeah, yeah, absolutely and I can, you know, maybe, um, folk a little about it. So I used to get a book, like this. Um, and this is like one of the, let me see if I can. It's not It doesn't if you hold it back and then down, there it is. So, I bet, I like this books, like it's, it's was an inspiration for the the work we're doing. It's called Intelligent, uh, continuous security. I don't know the author, this is not a blog, just a book that I I enjoy. Um, but the way you read, I think I benefit from technical books is I need to define an outcome. Like, what is it that I'm trying to do? Like the brain, you need to orient your brain. The brain needs to know, like, "I am going to New York," and I then need a map to New York. And as you're driving, you look up where you're at and what what where you're going next. So, this is how I use books. Just because reading it cover to cover, you just don't retain anything. But while problem-solving, learning like when there's an effort and it's hard, you tend to retain your information. So if the goal is to retain the inform- return the information in your head and be able to recall it in the future, don't try to recall, like, the those tactics don't work. Um, start thinking about the project that you would like to deliver, get the book that will assist you in to building that. Maybe skim through it, don't read all of it, you know. Then you say, "Well, I need a function that validate, you know, user input," or, "I need a function that validate the password." I do not know how to do that. I do not know where to put the password. Is it in a database? Like, how can I do like, should I hash? What should I do? Just look up that part, get it, that's it. Step number two, "I need to do I don't know, like, the back-end API for processing images and uploading them to S3." Never been Right, keep it very practical, like, chunk it out. Yeah, and eventually, you find that you got what's you need from the book. Maybe you didn't cover all of it, it's fine. But you got a lot out of it because you were able to use it to build something and once you use your brain to build something, you tend to retain that information for a long time.

[25:35] Jenn Gile: Okay, last question. Um, you told me you're hiring. What are you looking for? What role are you looking to fill? What is a good candidate for that role look like?

[25:48] Ammar Alim: I'm actually looking for a machine learning engineer. So, yeah, we would like to bring some, uh, person to be full-time, um, on the WAF project, to help us build those kind of models for our customers internally.

[26:07] Jenn Gile: Very cool. And are you looking for someone with, um, experience having done that before, given this is all a little on the new side, or are you more focused on, um, attitude?

[26:19] Ammar Alim: It would it would be a plus if we find someone, um, who have been there, done that. Those people are not a lot, so, um, we are very flexible with something that we did something adjacent to this in any space. It's a data problem and the machine learning concepts do carry over to anything with data.

[26:44] Jenn Gile: That makes sense. Well, this has been great. Thank you so much for taking the time.

Resources mentioned in the video

• https://careers.adobe.com/us/en/job/R158729/Sr-Machine-Learning-Engineer-WAF-Detection
• https://www.oreilly.com/library/view/ai-engineering/9781098166298/
• https://www.udemy.com/course/the-complete-agentic-ai-engineering-course/?couponCode=MT250915G3
• https://www.udemy.com/course/llm-engineering-master-ai-and-large-language-models/?couponCode=MT250915G3