Demystifying DevSecOps: Insights and Strategies with Tanya Janca (Semgrep)

Hello everybody and welcome back to Build Amazing Things Securely. My name is Laura Bell Main. I am your host and guide in all things awesome technology. And today I get to be, I've got a hero with me. Don't tell anyone. Don't tell her she's going to get weirded out. But today I have the beautiful, fabulous, incredibly talented and smart Tanya Janca with me, who you will know from the internets, but I'll let her introduce herself. And it's an absolute pleasure to have her on the show today. We're going to be digging into lots of cool stuff about DevSecOps and notably where it goes wrong. But before we get started, let's make her feel welcome. So welcome Tanya. It's amazing to have you.

Hi, Laura. It's so lovely seeing you again.

So we don't record the video on these things. So the lovely audience at home can't see how fabulous Tanya looks today, but she's really rocking it. So for those who haven't encountered you before who are you the human?

am a person that's fairly obsessed with creating secure software. I as a software developer I started coding as a teenager and got my first job, like the moment I turned 18 writing software at Nortel in Canada. That's the accent you hear. And then after 17 years of that and college and starting my own companies and doing stuff like that, I decided to join the dark side. And by that, security. And I started speaking at conferences so I could get in for free. And I still do that, so I get in free. And that turned into this really weird job that I have. So I just started a new job at a little company called Semgrep. And I basically run their online community. In their community events, and I give training and so some of the trainings for the public, a lot of the trainings for customers, and so I just have this really weird job where all my technical skills and my extrovertedness come together in a really nice way. And so I didn't know such a job existed.

That's amazing. And just sharing a little bit about that whole getting into conferences free from speaking. It is a totally a valid game plan. If you do want to see the world, there was a running joke for a long time that it was called the holiday club, where you could if you worked hard and did good talks that you could pretty much go anywhere, it seems to have worked really incredibly well for you, Tanya. And, what an incredible background. And I love how you call Semgrep a little company. I'm sure many of our audience have come across them. So if they're little, then many of us haven't got a hope. But let's get into it. We. We've been known, we've known each other for a little while now, because we we did similar things, but in different hemispheres. The audience knows I run training, you do training. We're both really fanatically obsessed with this idea of building amazing secure software. But we've both seen some horror stories on the way. And you've been doing some fabulous conference talks lately and I caught the recording of one of them recently about anti patterns or what goes wrong in DevSecOps. And I thought, it would be fascinating for our audience, who are mostly software engineers, not from the security world to talk through some of those today. So let's talk about how did you end up spotting so many of these negative behaviors? Where did this intel come from?

So I have this weird side job. So when I started speaking at conferences, Laura, I realized really quickly if I didn't do any AppSec, I wouldn't have things to talk about. So I started some long term coaching contracts, where I would just do like DevOps two hours a week with this one company for many years. And so I got to build lots of like long term projects, which was really cool. And then I also started doing... I joined the faculty of a place called Ayaan's Research. And basically, I don't know if you remember from the 80s, 90s, there was this television game show where you could phone a friend, and they would give you the answer to the thing.

Yeah, and you hoped you had smart friends, yeah.

so Ayaan's Research, you get to phone a friend, but the friend is me, or Dave Kennedy, or like all these different famous and less famous people, but people who have worked in it for 10, 20, 30 years, and we just crush your problem. And so I literally actually just got off an iance call with a company and we talked about, they acquired a company and the company's doing DevOps, but they're doing. Low, slow waterfall, and they're trying to put the two cultures together. And it's like, how do we help them not kill each other? And also actually release more secure apps and not have everyone be jealous of everyone else. And,

That's a lot to jive on a single call. That's a lot to conquer. So yeah. Kudos.

was a lot. And so then I looked at my stuff. So basically in January I had to rewrite one of my conference talks because I no longer owned the intellectual property that I had submitted to RSA. And I had two weeks to rewrite my three talks or be rejected. And so I spent two weeks of writing and I looked and I'm like I've met with over 400 companies now. That's a lot of AppSec teams. And they're... Big companies. All their names or almost all their names.

Yeah.

And And so I looked across all my data and all my notes. I'm like, here's The 15 things I've seen but from at least 10 separate companies and some of them 20 30 companies Repeating these same mistakes and I and so now when I do calls I'm like, oh watch out for this and oh make sure you don't do that and oh blah blah blah and they're like Oh, this is so helpful. How did you learn this? I'm like, oh from the 300 companies before you and so I summarized it all and then was like people want to just hear what not to do. And the RSA folks said, yes they do. And so I turned it into this talk and you were saying, anti patterns, like patterns that we know for sure every time they're bad. And I really like the way you worded that because I usually think of that as like regular expressions or code and not behavior. But yeah, there's anti pattern and I like the way you word things, Laura.

That's all right. You can steal it. I won't tell anyone. alSo okay. So we've got this whole wealth of experience then that we need to get out of your brain in the next 20 minutes. No pressure, but we can do this. So if you were to pick a few examples of these, what have you seen go wrong most commonly? I'd love to hear some examples and perhaps talk through, how on earth does that happen? How do we avoid it?

Okay. So the number one, one that I see the most often is just breaking builds on false positives. And it's almost always promises made by sales teams at vendors. Where, like, so I work for a vendor now, and I'm always

We forgive you

truth. It'll be great. Thank you for forgiving me. I was thinking of getting a shirt that says vendor scum.

Yeah. You'd still rock it, so you know, we'd still have to love you, so it's okay.

But basically, they buy this tool, they've paid Often, five figures, if not six figures for it. They believe it's going to help them with everything, and they just throw it directly into the CICD. They don't run some manual scans first, they don't, or maybe they do on only one app, and then they just start throwing it into every single pipeline, and then of course it... And they set it to break builds. They don't just put it on the learning, which would be a great way to start, or it'd be a second, less bad way to start. And they just, they're breaking builds for the wrong reason. Like it's not. broken. Like it's fine. It's a false positive. And what I like to do instead is first I scan one app manually and I figure out what's true and what's not. And I tune and I improve until I'm like, okay, these are legit issues. I get, them fixed. And then I work with them and I'm like, Hey, I could put it in alerting mode. And we check out that. And then it's still not doing false positives. And then eventually maybe we turn it onto breaking mode, but maybe we don't need to, because maybe socially, But They accept, I should not put highs in prod. So it depends on the environment, whether you actually even need to break or not, but so if you could test it out, and so usually I test with one team that likes me already, and then I expand to more teams, and Then eventually I'm like, okay, so I understand what triggers most of the false positives, and then we start rolling it out everywhere, and again in alerting mode. A few months of alerting mode, it's like you know what's wrong now, you should have fixed it by now, maybe we'll flick on breaking, maybe we won't. But I'm reviewing reports, and if I'm seeing a whole bunch of highs out of a team, and they're still pushing to prod, it's like, we have discussions. That we need to have but this is the number one thing I see the most often But there's there's a bunch more Laura some of them are the audience So I've given this a few times now and I've found it interesting the audience It's picking on different ones I didn't know other people were doing. So I, when I call impossible service level agreements or SLAs, it, so this might sound odd, but I've worked with, so I'm in my mid forties. I don't want to call people old cause other people call me old, but the old guard. So people who do things the old way where they want everything to be perfect instead of just quite good. So I aim for, we are too expensive, annoying and time consuming to hack. I should move on.

Yeah.

for most

Make them cry. And if they still wanna come, then you know, we can deal with it. But mostly just make them really sad and wanna go away.

So if that is good enough for, and that's your risk profile, that's acceptable, then I do everything at that level. But I've worked with people, Laura, where I worked at this one place and they showed training videos. And I'm not talking about WeHackPurple, but WeHackPurple, we have basically the exact same threat model. We show cool videos. There's copyright on the videos. We want to make sure you actually have seen the videos. If we're going to give you a certificate of But it's not some deep intensive exam. It's not like we're proving that you're good enough to be like a doctor or something important. It's just yeah, actually watch the videos and you did the thing. So I was doing services for another company where that's their exact model. And the guy was like we can never release a high or critical endoprod. And I'm like, yeah, we should not release new ones for sure. And he's no, we can never release any. So unbeknownst to me, what he was doing is he was blocking. So let's say there's an app and there's 300. Highs. I kid you not, Laura and like 50 criticals. And so me or some of the devs would push a whole bunch of fixes. Let's say eight criticals and five highs. And we would go to push it to prod, and he would block it. Oh, you're still putting 200 highs in, so no. They're already there, dum. We're reducing the number. And he's you will not put any highs into prod.

Yeah.

kid you not, Laura. So I've seen this over and over. And so in your modern tools, you can have two service level agreements. And that's how I solved it, is no new high criticals or mediums. you still are allowed criticals, highs, and mediums that were present during my first scan. And then in the next six months, we tried to fix all the criticals. It ended at some of the teams didn't make the six months. Some of them had a lot of work to do, but most of them made it. And then we started breaking on old criticals and new still medium high critical. And then the plan was the next year, although I got acquired, so I'm not there, but they're working on it still. They're chipping away at the highs when I left them and chipping away quite nicely But we would still battle with him breaking a build saying no because there's still five highs Yeah, but there were 20 and now there's only five and they're the five same ones that are in Prague right now We're not having and he was just like You don't understand quality, Tanya. And I was like, that's funny because I have lots of comments for you that are less polite. But I did not save them.

It's a delightful thing about having a Kiwi British person and a Canadian on a call like this that we can say things to us as scathing, but everyone else will be like, why are they so polite? Look, there's so much to unpack here. I'm just going to pull some bits out because I think our audience are going to want to pay attention to this. So when we're talking about that first anti pattern, there's no magic boxes. Like it doesn't matter what the vendor told you it's going to do. It's not going to work on day one, out of the box with no configuration. And I love that idea of really easing into it. So running it manually for a while, running it in alerting and then moving on to breaking the bill, because how many times do we build a day now? So many, and if it's breaking every time and in ways that are really not that important to you right now, yeah, it's going to really annoy everyone. And yeah, you're the pragmatic approach there of, yeah, we've inherited other people's code a lot of the time, that code comes with problems, not just security problems, but a whole tangled mess of history that is tied up in our code base. And you can't fix it all in the new job. You can't come in and say, Hey, no, we'll draw a line right now and everything's gonna be perfect from now on. So that's really comforting to hear that other organizations are doing it. You know that it's a problem elsewhere. So if you are listening at home right now and you're like, Oh yeah, that's us too. Hopefully you'll feel comforted. This isn't just you and that there is some kind of pragmatic middle ground of working your way towards that better state. Oh, so much gold here, Tanya. So much gold. What else have you seen? There must be horror stories.

So another one that I, so I call it Insecure System Development Lifecycle or SDLC. And so this is where they're like, we have bought the three tools we were told to buy. We have shoved all of them into the CICD, not necessarily nicely. We are breaking builds with no abandon. And the entire rest of our system development life cycle has zero security, zero support, zero training, zero people that you can ask for help. And then we're wondering why it, they're not passing any of our builds. I don't understand. And I'm like, okay, so when you kicked off the project, did you explain who their security resource person was? They're like, no. I'm like, okay. Then when you were doing requirements, did you give them any sort of security guidance or requirements? No. When you were designing, did you review their design? Did you offer them any patterns that you might want them to use? Did you do a threat model? They're like, no. I'm like, okay, so then coding, did you give them anything to use in their IDE? Did you give them secure coding training? They're like. No, we bought the tools, we wrote the checks, we want secure code.

Yeah, it's that gym membership problem, right? Where you spend your 30 a month on a gym membership and you hope the light of fitness will shine upon you and you will suddenly emerge, a gym bunny. But in reality that's one tool that you need but it only works when you've got behavioral change all around it. And I think, the one thing that I take from stories like this is if we look about at the, when we build code by that point a lot of decisions have been made already. Like we talk a lot about the syntax that we write, but by that point a lot of it's set in stone. And it seems such a shame to have such a big gap in your process where you've got so much effort going on, so many wonderful people creating wonderful things, but not adding that little extra ingredient just so that by the time it comes to coding that it's all there. Cold.

Yes. Honestly, Laura, a lot of people are like, Oh, we're supposed to do that? Or, how do we scale that? There's a lot of conversations about how the rest of the SDLC could also be secure. And then so another conference talk I've been doing a lot this year is called shifting security everywhere, because we constantly talked about shifting left, which means starting security, in my opinion, at the very beginning of the system development life cycle, but it meant making it earlier. So don't come in at the end with a pen test. As your first security effort. But then it turned out, Laura, people released the stuff into prod and they're like, Cool, I'll never talk to you again. I don't need to look at you. You're perfect. You'll clearly age perfectly and never have any problems. I don't need to monitor you. I don't need logging. I don't need to like do instant response or be ready for that in any way. And so it turns out I needed to talk to a lot of teams about, okay, so your software that's already in prod still has security needs. It still needs loving and caring and feeding and all of those things. I'm like, are you monitoring or logging like anything? And I know why we log the. We monitor the network and I'm like, don't you care about your apps too? Your apps are on your network. And there's Oh, and so I have had a lot of discussions this year about after you release, but I'm off topic, I'm sorry.

No, you're not. This is, it's wonderful. And I have small children. So when somebody says, and that, yes. And then what? Like my head goes to Olaf from Frozen. So I clearly, my, my brain is in a weird place today. But I think this is all really important. And I think there's a few things that we focus on. We call it on the safe tech side code that's not currently being built. So if it's out in production, you're never running your build tools anymore. It doesn't get built anymore. All of those updates to the signatures that are coming through on your new shiny tools You've paid a lot of money for you're never getting to see that against your code because well, it's just that they're doing things so If you were to make wave your magic tenure wand and make the world a better place and give our audience some advice What? What are the highest priority things you would recommend a team who's either starting out or they've got something that's not working to do for, to make the security of their software better?

So there's two things I think every team needs if they don't have them, and one is security training. And by that how to do your job responsibilities securely. So if it's your job to write code, how to write secure code. If you're a software architect, how to make sure that what you're architecting is secure. Whatever the job is, it's how to, so not don't click this link, we need those security awareness generalists. Things are nice, but if you have a sysadmin, they are an expert in many ways, right? They need to be very securely doing their job because they have so much power, Laura. They are very powerful. And I remember when I was a dev, I did not understand how much power I had. I was like, oh yeah, then I just grab the thing out of the production database, and then I run it through our decryptor, and then I just read them their password over the phone.

Oh,

much wrong there.

so if there's layers are wrong.

did. Yeah, I did that 10 years ago for sure because I didn't know all the there's so many layers of wrong listeners

You're hearing this because what's lovely about this is, everyone assumes that folks like us who've been doing security a long time because we have our knowledge sparkles in our hair and we're a little older than many of the audience that we must have just been born this way. No we're here because of the mistakes we've made on the journey because of the things we've learned. So yeah, I love that you're sharing that. We don't judge. We love that you came back from that and built an amazing career around it. So what else should we do?

best way. Okay, so one more. So the thing I end the conference talk on. is hiding our mistakes. And this literally and figuratively. So I've worked at places, Laura, where Okay, so I personally have done this mistake. And I remember I had this boss named Phil, and he with this team of devs that would not listen to me. They'd literally say, F off, Tanya. Stay the F away from my devs. I don't have time for your crap. Stop wasting your time, etc. And he would not let me scan their code. He wouldn't let me do anything. And so my boss Phil was like we have to take action. And I was like, you're going to go yell at them. And he's no, I'm going to talk to them and tell them how desperately we need them. And I'm going to tell, show them all our dirty laundry so they can see how pathetic we are and that we can't survive without them. I'm like, no. Do not admit weakness. No. And he's I'm doing it. I'm like, please don't do it. He's I'm doing it. and he explained three recent security incidents that I'd responded to. And the first one was where a key logger had gotten onto some of the computers. And he's one of the people was a grandma and she works in her, like for our company, a lot of you have met her. And we had hundreds of people, so it wasn't clear who it was. And he's she got so afraid, Tanya's teaching her how to change her bank passwords and stuff, because she had done banking and all these personal things. She started crying. So Tanya had to hug a crying grandma. That's what happens when we fail. Grandmas cry. He's this is serious. And then he said another one, and then he said the third one, and the third one was this... Software that had this vulnerability, and the vulnerability cost us more than my house that I was living in at the time. That was how big the incident was. It was quite terrible. It caused serious damage. It was humiliating to our organization. We had to report a breach to the government. It was so bad, Laura. And they're like, that's terrible, and he's that was your app. That is your app that you wouldn't let Tanya test. That is your app where she tried to talk to you. I remember her trying to book meetings with you over and over again, and you told her, and I quote, F off. Stay the F away from the side of the floor. We don't need you to waste our time. And they're like, and he's She can't do this without you. We can't do this without you. We make huge mistakes that hurt our organization if we don't have your input. And he like just begged them basically. And their boss completely changed 180. And he went from go away, literally putting his hand up like this in my face to like walked up to me and he put his hand on my shoulder and he's I had no idea how serious this was. And this will never happen again. And he like led the charge. It was like, that scene in Braveheart where Mel Gibson has like blue on half of his face and he's riding around on the horse and he's like freedom. It was like that. He was so passionate and excited. And he, I swear they like got imaginary horses and rode out of their like Calvary. It was like, So intense, the feeling from that meeting and my boss was like, it takes strength to show vulnerability.

Oh, that's so wise.

how are you so much wiser than me, Phil? he like, so he taught me that day that showing some weakness can actually be a thing that people that are very strong, you have to be strong to do it. And so since then I've started trying to share when I'm allowed. So I'm not always allowed, right? The only reason I could share this was because there's over 400 clients that I've served and they have thousands of clients. So it's unclear. from the thousands and thousands of clients they have, which ones these 400 are, and obviously, which, 10 to 20 had whichever specific issue. But normally we can't share. And when, as an industry, we don't share these lessons, we all trip over the same mistake over and it costs not only thousands of dollars and competitive advantage, but it literally results in grandma's crying in my arms. It results in people losing their jobs because we spent 400, 000 responding to one security incident that could have been avoided, right? There's layoffs happening all over the planet right now, especially like in the past 12 months. Imagine if we could have stopped a whole bunch of security incidents from happening and we could have laid off two or three less people. Those people's lives are completely changed when they lose their livelihood. And our industry is so secretive. And it's not making us stronger.

No.

so the last message is we have to share. Sorry to talk over you.

No you're quite all right. The passion is very clear in this. And it's such an important lesson. There are no aspects of our life where keeping it to ourselves entirely is the best plan. Now you may not audience at home be the type who speak at conferences and stand up and, rock thousand person audiences. But even if you can find a way to share it internally inside your organization where the trust level should be the same, you know between your security team and your dev team, they should be you're all in the fight together. Then sharing your own examples is really good and if you want to see some examples of this from the non security space go talk to your QA teams because this is a lesson that QA learned 15 20 years ago when they realized that if they weren't sharing what they found you couldn't find other instances of it you couldn't look at the real breadth and scope of an issue. Oh, so many things for our audience to unpack. So those listening at home, I'm going to give you bits of homework because I think that's the best thing to do here. I'd love you to go look for magic boxes in your environment. If you think you've got one, go play with it a bit, run it manually, go do some scanning that doesn't break your build and annoy everyone. Go create some connections, go use that social capital and show some vulnerability internally and see where that gets you. And check out your service level agreements. It may well be that the best thing you can do is allow some imperfection so that you can enable progress forward rather than just hitting a wall. Oh, so much gold here, Tanya. You've just started a new adventure. You're, you're now part of the Semgrep family. And that's amazing. Congratulations. I'm sure most of our audience know you already, but for those who are just encountering you for the first time, what's the best way to keep in touch with you and see what you're up to?

iF you go to shehackspurple. ca I have all of my links and everything there. So I have a blog, I have a newsletter, I have... I don't really love email. I have all my social media, like my Twitter and my Tik Instagrams and all those things. But I feel like if you want to know everything that's happening, if you join the newsletter, you'll get basically once a month, all of the content I've done and every single event I'm going to be at with invites and free stuff. The other thing you could do is you could join. either or both the WeHackPurple community by going to community. wehackpurple. com and or joining the SemGrep newsletter and I'll put a link for the SemGrep newsletter for you but basically slowly WeHackPurple is melding into SemGrep but WeHackPurple is still slightly more active with more stuff so you can join both or just join Like the newsletter and you'll get invites to all the new things as we basically take both platforms and move them into one bigger better platform, which is gonna take a few weeks, unfortunately So I guess newsletters is what I have for you right now, but soon there'll be more

we've gone old school newsletters are the way forward of the future, and I love it. And, I'm gonna call it out with the audience. 'cause, I run a training company that directly competes with what Tanya does. And the reason that we're talking together, the reason we're friends, is because it doesn't matter. It doesn't matter whether you choose awe or you go to Tanya's sessions or you go to both, or it doesn't matter. The aim is for all of us is to build amazing, secure software. And so it's an absolute privilege to have peers out in the world who are fighting the same fight. And thank you so much for sharing your knowledge with us today, Tanya.

Oh, thank you so much for having me Laura And thank you for all the hard work you and your team do. You move a lot of things forward in our industry And I appreciate you

Awesome. That's why I'm tired then, okay. So those who are listening, the drill by now, you need to like, and. Subscribe and click a thing. You could comment on a thing. It could even be this thing or someone else's. It'll make someone's day. And remember, if you're looking to just do one hour of security every sprint and make your world a little bit more secure, then sign up to one hour appsec@onehourappsec.com and you will get yet another newsletter with just 60 minutes of activities you can do to bring some AppSec to your lifecycle, however big or small it is. Thank you so much for joining us team, and we will see you very soon.