Episode #001 - Databases and DevOps with Silvia Botros

About the Guest

Silvia is a Principal Engineer at SendGrid, a cloud email provider with household name clients like eBay, Spotify, Pandora and Airbnb. She is an avid distributed systems and databases tester and spends a lot of her day trolling her Ops team. You can hear more from Silvia on her blog https://blog.dbsmasher.com and on Twitter at @dbsmasher

Key Takeaways

Involve your DBAs in the engineering process early on to prevent problems in the future.Enabling engineering teams to manage their own schema is critical to reducing day-to-day toil of the DBA team and empowering Engineering.If you’re wary of automating your databases, set up guard rails to add some safety to it first.

Links Referenced

SendGridAndy Jassy’s 2018 Keynote at AWS re:InventStories From the DBA Trenches - Silvia Botros (Velocity London 2018)Database Reliability Engineering by Laine Campbell and Charity Majors (affiliate link)SendGrid blogState of DevOps Report by DORATool: slowquerydigest (pt-query-digest)Tool: AnemometerTool: ProxySQLTool: VitessAuditing Databases at The Grid by Silvia Botros

Transcript

Mike: Running infrastructure at scale is hard. It's messy, it's complicated, and it has a tendency to go sideways in the middle of the night. Rather than talk about the idealized versions of things, we are going to talk about the rough edges. We are going to talk about what it's really like running infrastructure at scale. Welcome to the Real World DevOps podcast. I'm your host, Mike Julian, editor of the Monitoring Weekly Newsletter, author of O’Reilly’s Practical Monitoring, and a DevOps consultant/analyst.

Mike: Hi everyone, I'm Mike Julian, I'm here with Silvia Botros, principle DBA at SendGrid. Welcome to the show, Silvia.

Silvia: Thank you for having me.

Mike: You work at SendGrid. There's a lot of people that have no idea what SendGrid is. Monitoring Weekly is actually a customer as well. I use it for all of my contact forms. It's wonderful. But for those who don't know, what does SendGrid do?

Silvia: First, thank you for being a customer.

Mike: Absolutely.

Silvia: I like to hear that. What we do is basically an email service in the cloud. A lot of companies realized years ago that they need as part of their engagement with their customers whether it's onboarding new customers or communication with existing users, that they need to send a lot of email. It turns out that sending email is a complicated thing. There's a lot of rules around it. SPF, whitelisting IPs, dedicated IP or not. Block lists that the inbox providers maintain. There's magics of times depending on the provider, whether email looks fine or it looks spam-y, or it looks fishy. Turns out there's a lot of logistics around that, that these companies all across the board don't really have the skills to deal with and don't want to spend resources on.

Silvia: Customers of ours include a large swath of companies large and small. Some of our big high profile customers include eBay, Spotify, Uber, Pandora, so you get the gist. You sign up for an account, you need a confirmation email. We have to click a link. They want that to come to the inbox, not to the spam folder and they want it to go to the inbox fast. That is basically the service we provide to them. We also now, as you mentioned have your contact list, we also have a marketing email, a marketing campaign's product that we provide to feed into this and provide the more advanced, the more geared towards marketing and that crowd. Not just the day-to-day transaction stuff.

Mike: Yep. Yeah, it is a surprising huge pain in the ass to actually manage all that stuff. I know several companies that just use their corporate email accounts to do it and surprise, surprise everything goes to spam.

Silvia: Yes. I will fully admit I did not know this was a thing, that this was a viable business model before I got my job at SendGrid. But once I started learning about all the details it dawned on me, how much it is ... Why would you build that yourself? If you are eBay, and you're focusing on bids or getting traffic or if you're Pandora and focusing on getting people to listen to playlists, why would you worry about spam lists and how every individual inbox provider decides how to judge your emails?

Mike: Yeah. That's a rabbit hole that seemingly never ends.

Silvia: True.

Mike: You're a DBA for them which is kind of weird. Those are still a thing? I thought when we all started rubbing DevOps on everything, that DBAs just kind of went away. We didn't really need them anymore because now the developers are handling it.

Silvia: I know, like we had thought DevOps were going to eliminate the sysadmin didn't we?

Mike: Right and it turns out we still need those, so I suppose we probably still need the DBA. What does a DBA actually do? What's your role now? What does it actually look like?

Silvia: A DBA in a company that is honestly trying to apply DevOps is practices, which is whole other rabbit hole we can go into. A DBA in this environment needs to be involved in setting standards of practice, that's helping engineers debug things early on. Helping design. A lot of what I do is early cycle, early life cycle of software where they're still building a new thing and I get to give the input on whether this thing has the potential of growing really fast and what does that provide as far as implications as to the data store. Is the data store they're using appropriate for the use case? A lot of guidance, a lot of things like that. There's also still a lot of day to day things in production. For example, we are SOC2 compliant and we are now public company so we also have to do cover Sarbanes-Oxley. There's a lot of compliance needs that don't necessarily fall into any feature work but still have to be covered because every year we have odd pairs, we have fiduciary requirements that we have to cover and so the database ops team, which is what we call my team right now, is responsible for doing the work that provides the evidence that we actually are compliant with all the things that we have to be compliant with.

Mike: That's kind of interesting. What compliance stuff actually surrounds the database? I know GDPR probably has a big component in here doesn't it?

Silvia: It does. That was definitely a big thing and it covers not just relational databases, it covers a lot of data sources so in practice it involves more than just the DB ops team. To provide context, currently my team is solely focused on relational databases so we have a pretty large MySQL footprint and that's where our focus is.

Mike: Yeah, every time I think about DBA's, I think about just relational databases but once I started thinking about that more and about you and I chatting, I realized...