0:00

Julie: When I am on my game, which of course is all the time—always—[laugh]. Jason: Welcome to Break Things on Purpose, a podcast about reliability and learning from failure. In this episode, Julie Gunderson and Jason Yee share their favorite moments with this season’s guests. How’s it going, Julie? Julie: Good. How are you, Jason? Jason: I’m doing all right. So, we’ve got some changes coming for the podcast, which I’ll mention with our listeners later, but before we do that, we’ve had a fantastic run for Season Three here, starting off with a fun B-roll episode about unpopular opinions that was, surprisingly a popular episode, more than I actually thought. Julie: Now, I think it’s important to explain to our listeners what a B-roll episode is—if they haven’t heard the “Unpopular Opinions” episode—that we ask our guests some questions that they don’t know about after each episode so that we can put those together in a very themed episode. Jason, did you want to talk about that a little bit more? Jason: Yeah, to be fair, sometimes, when I’m on my game and I remember to prep, I will [laugh] actually send these questions to them beforehand. But the idea that we had was, well, it’s fantastic when we get guests on and we have a whole episode where we have these fantastic conversations, the B-roll episodes are really fun because we like to ask our series of guests the same question after we record the main episode, and then we kind of stitch them together. And so, back in January, we had an episode called “Unpopular Opinions” in which I pulled out a few unpopular opinions that some of our guests had and they talked about them. So, we had three guests on for that episode. And I think my favorite unpopular opinion on that was actually from Season Two. So, we had this guest on in the “Unpopular Opinions” episode named Brian Holt, who is a good friend of Pat Higgins—who was a previous host—and this is from Season Two. And Brian had this great unpopular opinion about React hooks, which was interesting to me because I am not a front-end guy; I don’t know React at all. And apparently, this is super-controversial in the React community, and so it led to a… I will call it a raucous debate in Twitter about [laugh] whether Brian was correct or not. Julie: Yeah, Jason. You really started something there with that. You’ll put a link to that tweet in the [notes 00:03:02] on this episode, right, so folks can follow along? Maybe— Jason: Yeah. Julie: —add their opinion. Jason: Yeah. We’ll [link back 00:03:07] to that episode. Again, if you’re searching for it as the January 11th, 2022 episode, but we’ll link it in the [show notes 00:03:14] and I will try to capture the tweet that started the whole debate. Julie: I do like how you’re a much kinder host than I am because when I am on my game, which of course is all the time—always—I still never send to my guests the questions ahead of time. But for those that are curious as to actually what our questions are— Jason: Actually, do we want to spoil that for folks? Julie: I mean, you should just be surprised because we clearly don’t want to spoil it— Jason: Yeah. Julie: —for folks. Jason: I mean because if you happen to be a guest on the show, then yeah, we want to just—Julie likes to just spring these on us. So, we actually won’t tell you. You can find out as we release the episodes, which is fun because then we don’t use that question anymore, so no tips on being well-informed. Julie: Because that helps you be more reliable. Jason: [laugh]. Julie: We should probably [laugh] move on though to the next episodes. What Jason is not mentioning on this episode is we recently did a lot of travel. I have not been back in the United States for more than 24 hours and my jetlag is very great today. So, we’re having a lot of fun. I finally figured out what day of the week it was; that was very exciting. Found that out earlier today it was not the day that I thought it was. But yeah, this year—or this season—we’ve had a lot of really great guests on our show and I’ve been honored to get to just talk to some of these folks. Jason: Yeah, I’m curious Julie, who are some of your favorite guests from the season. Julie: I mean, all of them. All of my guests, of course, are my most favorite guests. But I mean, [laugh] we’ve just had a—we had the one episode—just even if we weren’t in order, like, we have this episode with Sam Rossoff, who’s here at Gremlin, and he talked about data centers inside of data centers, and this, like, major incident that they had at AWS. And that was just kind of an amazing episode. And of course, we talked about Nintendo, which any episode that starts out with talking about Nintendo has got to be a great episode. But that one was really excellent. You know, Sam talked about his time at Amazon and then his time at Snapchat, when Snapchat had, like, 15 engineers and they went to, like, a thousand. We covered, actually, quite a few things on that episode, so I think it was just a lot of fun to record. Again, we went from everything from Nintendo to NFTs. Jason: The other interesting thing about this is, we normally don’t have a lot of Gremlin folks on the podcast, right? Because we particularly like to get a lot of different viewpoints and see the world—the technical world—outside of our own little ecosystem or our own little company here at Gremlin. But we actually also had Natalie Conklin on. Julie: We did. That was another amazing episode if I do say so myself. But Natalie is the Head of Engineering here, at Gremlin. And this was a really exciting episode because we recorded it a couple of weeks before she gave her closing keynote at DevOpsDays Boise. So, Natalie actually flew out to Boise to close out our conference. And this episode is titled, “Embracing Change,” and that actually was part of the title of her closing keynote at the conference. But Natalie talks about, like, embracing change fearlessly and she’s just got this amazing career as a woman in tech, as well. And just great stories to talk about, and how we can really all learn and grow and we shouldn’t be afraid of failure. And really, how do we embrace that change? So, really exciting to have her on as well. But how about you? You also recorded some episodes. Jason: Yeah. Yeah. You know, one of my favorite episodes, actually—because there’s some backstory to it—is the episode with Gunnar Grosch, who’s currently an AWS Senior Developer Advocate. And the reason why it was so exciting for me is because Gunnar has been in the Chaos Engineering community for ages. Like, I mean, as far as I can remember, he’s always been involved in some way, talking about Chaos Engineering. And in fact, I knew him before he joined AWS, and so we had chatted about actually getting him on the show. And it was always this difficulty of just trying to align our schedules. And then he joined AWS. And for folks who don’t know, AWS, it’s a large organization, which means there’s a lot of restrictions about official communications. And so, he joined, which was fantastic, I mean, he’s been a community leader in AWS technologies for a long time, so it’s great to see him get recognized and actually hired by AWS to continue that work, but that immediately meant that he couldn’t be a guest until he actually got, like, official permission to speak on behalf of AWS. And things had just been going on for well over, like, a year, year-and-a-half, like, just trying to get him on the show. And so, we were finally able to do it, which was absolutely amazing. So, it was great to hear his story about getting involved in Chaos Engineering, and what he did early on, and what that looks like now. Julie: You know, when we go back to talk about the folks that, like, started early, that reminds me of the day of darkness episode that we did with Chris Martello over at Cengage because Chris had been doing, like, the QA side for a while that he actually started out as, like, a high school science teacher. And, you know, when we talk about experimentation, and how Chris started out to where he is now, and he’s a huge advocate of Chaos Engineering. And I just love the fact—and I think I said high school, but he was actually middle school science teacher. So, he had been, you know, playing around with Chaos Engineering really, for quite a long time. But, you know, according to you, Jason, the way you talk about it which I always love, is you talk about how, you know, we’ve been practicing Chaos Engineering since you know, like, the beginning of man, right? Jason: Yeah. Yeah, I always say that Chaos Engineering really was the first person that ever existed that was like, “Huh, I wonder if I did”—and then you fill in the blank with some risky kind of ridiculous behavior. And so, they went and did it anyway, despite probably risking, like, personal health and safety. Julie: But that’s where we’ve learned to grow. I mean, that kind of translates into Dan Isla’s episode. So, Dan was at JPL and we titled his episode, “Astronomical Reliability,” which anybody in, like, the physics space might understand the astronomical numbers, but Dan was talking about how embracing failure is great and lovely and we all want to work towards that, but it doesn’t work the same when you’re sending a rocket into space with people on it. Jason: Yeah. Turns out that with engineering, there are a lot of constraints and a large part of the job is actually understanding the constraints that you’re working within. And so yeah, I loved Dan stories about some of those constraints when basic things that we come to rely on, like, NTP, right, getting time synchronized across servers just literally doesn’t work because you’re, like, literally not on the planet. Julie: Yeah. And they found a drift of, like.000 seconds, or however tiny, and that would have meant that the Curiosity rover would have completely missed Mars, when it was just, again, astronomical numbers. And I really love, too, that Dan talked about—he talked about a little bit of his experience over at Google because he did go from JPL to there. Gave some great career advice in his episode, as well, about staying at companies too long, and—because he was at JPL for eight years, but then yeah, he went over to Google and then went over to itopia to work on the Selkies project. So, I think there was a lot of good stuff there. And, you know, you go from somebody like Dan, who has been doing this for an exceptionally long time and has a great career with a lot of diversity in it to Carissa Morrow, on her episode learning to be resilient because Carissa is very new in her career. So, she’s really been working as a developer just for three years, and we got to hear a little bit about her journey from learning and how it was really about others working to support her and help her and teach her. Jason: I really liked that episode, particularly because that was a new perspective for our show. Our show in the past has largely been a lot of really established engineers. And that’s something that we see a lot of in the SRE role, right? Because being a good SRE, oftentimes isn’t so much about your technical abilities, it’s just about those collected experiences and having seen problems before. And so, it was really great to see somebody that was—hadn’t been around for years and years and had decades of experience and all these great stories about, like, the crazy time back when, and then fill in the blank on the, you know, the contextual constraints that we’ve had way in the past. It’s fantastic to hear about how folks are entering this field, brand new, and how we can actually use her experiences and some of that insight from new people in our field to make it more diverse and more open, and more inviting to new people. And ultimately, that’s what we need because we need to grow this if we’re only relying on veterans in technology to be SREs, then we’re always going to be just this tiny little sliver of technology and it’s always going to be problematic of not being able to build teams that are big enough to handle, you know, systems at scale and, you know, on-call rotations, et cetera. Julie: Yeah, and it’s a good reminder too, that how more experienced people treat newer people can completely shape their career, you know? When you tell somebody that you believe in them and you believe that they can do something, and help them learn how to get there, you’re building those engineers of the future. One thing that I thought was really fun with her is—and kind of translates across a lot of our episodes—is that first what she called a humbling experience, that experience where you break the thing, like, really bad, the thing in production. And I’m not going to give it away, she talked about it on the episode, so definitely go take a listen to that, but we’ve talked about that with others. And everybody remembers that first experience when they took down production. Jason: That first experience really sets the direction and the tone I think for a lot of careers, right? Because you realize, number one, that there are some pretty big stakes at play. When we talk about reliability, you know, oftentimes we were just thinking, oh, well, it’s a computer system, things break all the time. But depending on where you’re working at even places that seem rather innocuous, like, oh, well, if the system goes down, you know, my customer may not be able to buy that the random junk that we sell, right, but it’s not, like, critical; it’s not life-saving equipment or anything. But ultimately, I think it helps underscore that when you do have that first outage, you can’t take it lightly what you do; it is important to keep systems running. And so, it sets some of the tone there. But it also sets some tone in when you fix that, and you actually get that feeling of, yeah, things broke, but because I’m here, because I’m doing this work, because I have this team around me that’s helping me, that we can actually fix things and make them better. Julie: So, that kind of reminds me of your “People, Process, Culture, Tools” episode that you had with JJ. Jason: Yeah, that was a great episode as well, right? JJ Tang, who started Rootly, we chatted a little bit about his experience. And it was that fantastic thing of, you know, we have these build episodes where we have guests on that have created some tool or some useful thing for us, and hearing his insight about coming from an organization where this was a problem, the whole incident response, like, there’s all this information you need to gather and trying to reduce that cognitive load, but ultimately, it comes down to that of having the right people, having a process, having a great culture, and then finally, lastly, is the tools, right? The tools, hopefully, help bring that all together and enable you to really leverage the people, process, and culture. Julie: Yeah, I liked that. And, like, one of our other great build episodes was the one with Elizabeth Lawler, which I love on so many levels, just because I know Elizabeth, I know her from her time over at [unintelligible 00:17:06], and now she’s gone ahead and founded Appland and Appmap. And that was another one where folks see a need and then build the thing to fill that need. And I’m always amazed when people can build these companies. I mean, I know, this is a Season Two episode, but you also had Itiel Shwartz on from Komodor— Jason: Yeah. Julie: —and they just got a huge funding round. And so, we get to see these companies, as they’re in these early stages and growing, and it’s just such a great testament to the hard work. And I always kind of think about, like, what would it be like to be a founder where you had this idea, and you probably whiteboard it in your garage, or on your living room, or maybe even sticky notes. And then all of a sudden, you go back and look and now you have a successful company, where you’re employing people. And, you know, eventually, you’re AWS or however Amazon started. Jason: Yeah. Yeah, I think that’s one reason that I really love those build episodes is we get to chat with folks and get some of that insight. You know, as I think about that, and maybe that’s an idea for another podcast that we should run is chatting with startup founders and talking about, like, how do you go, you know, from an engineer—which is, I think, as I look back through the catalog of episodes, everybody that has been on a build episode has been an engineer, and most of them have done that of, they were an engineer; they built something that mostly was for themselves to be useful, and they found that others had found a lot of value in those tools as well. And so, they sort of stumbled into being a business person and running a company. So yeah, if anyone wants a free podcast idea, go ahead and grab that. I guess it’s somewhat similar to the How I Built This that NPR runs. Julie: Oh, well, I need to listen to that. But on the building things, we have the episode—“The one with.” I need to start saying that. I think that’s a whole different TV show’s way of naming—but [laugh] the episode that we had with Kolton Andrus, one of the cofounders of Gremlin, and Alex Solomon, one the co-founders of PagerDuty, and we actually did the crossover episodes. Just to let our listeners know, if you haven’t listened to that, the first part of the episode is on our podcast, the Break Things on Purpose. The second part of the episode is on PagerDuty’s podcast, Page it to the Limit. In that episode, Mandi Walls over at PagerDuty and myself hosted Alex and Kolton talking about you know, going from having an idea to founding a company. They both had spent time over at AWS and so they knew the stories there—or Amazon, I guess, depending on the areas that they were in. But just talking about their pasts and how they founded companies, and then actually how their paths crossed, and their learnings from each other, I just thought it was a really fun episode. Jason: Yeah, that was a great episode. Interesting to really hear how many similarities they’ve actually had in their experiences. Julie: Yeah, just like where they saw a need. That’s one thing that actually I thought was pretty cool. You know, Alex saw a need to revamp the whole on-call process and tools, and then he formed PagerDuty. And then I think what Kolton learned for Chaos Engineering and being able to take those reliability practices and make them accessible to companies that weren’t Netflix, that weren’t Amazon, and bring them out to the rest of the world of companies, I just think that, again, it goes to seeing a need and filling a need. And I think that actually translates across all of our episodes because no matter where people were in their career, there was always something that they were learning, or passing on. Jason: It’s all about that continuous learning. Well, with that, just to wrap up this episode, we are coming to the end of what we’re calling Season Three. There are big changes in store for this podcast and the media from Gremlin as a company. So, if you haven’t subscribed or you’re not following the show, I highly recommend that you do that; click the Follow button. Or you can follow us on Twitter at BTOP—B-T-O-P—pod—so @btoppod—on Twitter, and we will let you know about new episodes. We’ll be taking a short break and coming back, and so want you to be aware of that. In the meantime, though, during that break, definitely, if you have missed some episodes from either this season or past seasons, highly recommend that you look into our catalog. We’ve had a bunch of fantastic guests. So, go ahead and listen to those and we will catch you sometime after the break. Thanks for listening. Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. Well, be taking a short break, so subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or wherever you get your podcasts in order to receive notifications when our next season begins. Our theme song is called “Battle of Pogs” by Komiku, and it’s available on loyaltyfreakmusic.com.