4.2.3

Richard Tape of the University of British Columbia runs a massive multi-site WordPress installation infrastructure with tens of thousands of individual websites. A change to the WordPress shortcodes API broke hundreds of them, and Richard had to get creative to fix it. The incident would keep him busy with hundreds of urgent support requests for days, and it shook his faith in the WordPress project.

Richard Tape of the University of British Columbia runs a massive multi-site WordPress installation infrastructure with tens of thousands of individual websites. A change to the WordPress shortcodes API broke hundreds of them, and Richard had to get creative to fix it. The incident would keep him busy with hundreds of urgent support requests for days, and it shook his faith in the WordPress project.

About Our Guests:

Richard Tape is a WordPress core contributor originally from Manchester, England and now living and loving life in Vancouver, Canada. Checkout his “rant” he published days after his difficult WordPress update. Follow him on Twitter @RichAsInRichard.

John James Jacoby is another WordPress core contributor who gave us an inside perspective on the WordPress core update that changed the shortcodes API. He is the founder of BuddyPress, loves puppies and currently resides in Wisconsin. Follow him on Twitter @jjj.

The Hard Refresh theme is by The Brow, the latest in a long line of beat projects from genre-bending producer Marcus Williams. His ability to blend the visceral headnod qualities of golden era hip hop with dream pop and indie rock has captured fans across the world. He has been a steady fixture in the Portland scene through the years and continues to remain versatile – these tracks can be uptempo and playful one minute, then gritty and determined the next, but always confidently crafted with a unique penchant for sonic detail.

Interstitial music was composed by Douglas Detrick and Wolfgang Amadeus Mozart, and performed by Casey Bozell. The episode was recorded, edited, mixed and mastered by Douglas Detrick, and produced by Douglas Detrick and Catherine Bridge. Thanks to Richard for sharing his story, and to John James Jacoby for lending us his perspective.

Transcript:
Intro:

[Catherine] More and more, our lives and businesses depend on internet technology. These are stories about the people who pick up the pieces when it all falls apart. Welcome to Hard Refresh.
[Rich] My team and I checked it all through. Everything looked fine, so we put it into production. It took maybe seven minutes for the first ticket to come through! …my whole site’s broken, nothing works…

Theme music drops

[Catherine] Richard Tape of the University of British Columbia runs a massive WordPress installation with tens of thousands of individual websites. A change to the WordPress shortcodes API broke hundreds of them, and Richard had to get creative to make things right again. The incident would keep him busy with hundreds of urgent support requests for days, and challenged his faith in the WordPress project.
[Douglas] Hard Refresh is a production of Rocket Lift, a web development company based in Portland, Oregon. I’m your host, Douglas Detrick, and I’m glad you’re with me.
[music]

Meet Richard Tape: What does he do?

[Doug] Here’s Rich.
[Rich] My name is Richard Tape and I am a WordPress developer for the University of British Columbia.
[Doug] …which I’ll refer to as UBC. UBC’s website is filled with the usual information for prospective students and anyone else who wants to know more about the school.  He also runs several separate platforms dedicated to teaching and learning with many websites for faculty and students.
[Rich] The biggest one we have unimaginatively titled “Blogs” which is blogs.ubc.ca and it has at the moment 65 thousand sites and about 40 thousand users.
[Doug] And there are more. The total number of websites and users here is massive, and Rich is responsible for all of them. Good thing he likes a challenge.
[Rich] I got offered the job at UBC and it was too good to turn down. The reason it was too good to turn down was because UBC has been running WordPress multisite for the best part of a decade on campus, and runs one of the largest multisites in the world that we know of.
[Doug] All in all Rich manages nearly 100,000 individual websites. The scale of the system is incredible on it’s own, but that’s not all that’s important about it. The websites are vital components of the business of the University. Sometimes students are assigned projects that must be submitted online.

[Rich] If a student says “I tried to submit this thing” then we have to actually be able to see what happened. To see whether the student is lying, which is perfectly possible…”
[Doug] My digital dog ate my homework…
[Rich] “…or, what’s more likely, there was a failure somewhere.”
[Doug] The student can say “I submitted this work…”
[Rich] “…It was absolutely vital to my degree. Where’s it gone?” And it’s quite stressful! Ultimately speaking, it can go to lawyers.
[Doug] Rich didn’t tell us about any lawsuits that he’s been involved in, nor could we find mention of any involving UBC, but it’s obviously something that he’s concerned about. Just because it hasn’t happened yet, doesn’t mean it won’t happen. So, it’s Rich’s responsibility to make sure that this kind of failure doesn’t happen, and if it does, that he has some way to find out what went wrong.
[music: theme fades out]

 How does Rich manage that many websites?

[Doug] There are lots of details that make running a single website a big challenge, so running 100,000 websites sounds, well, impossible. But, It turns out that it is possible with the right tools. Rich uses a special feature of WordPress, called multisite, to make the job more manageable.
[Rich] “So, if you are relatively familiar with WordPress in general, you’re probably used to having just one website where you go to the dashboard, and that’s your WordPress dashboard. WordPress multisite allows you to run multiple websites from the same code base.
[Doug] Like from a single dashboard. One of the most important parts of maintaining a WordPress website is to keep everything up to date, including WordPress itself, your theme, and your plugins. With WordPress multisite…
[Rich] “You can have five websites running from one installation of WP. So, you can have one plugins directory, one set of themes, so that’s only one code base you have to update, when you need to update WordPress or plugins for themes for all five websites.”
[Doug] The collection of files, including WordPress, its themes, and plugins, is what Richard calls the code base. It’s what that the server uses to deliver the content of each website to you, the user. To understand how multisite works, you need to know how they work together. So here’s an analogy, with some pretty sounds too! You can think of the multisite system as something like a violin.
[violin sound design starts]
A violin has a single body—the part that’s made of wood where the sound resonates—and four strings. Each string depends on the underlying infrastructure of the violin’s body to support the strings’ tension, to resonate against and amplify its sound. So, each string is like a different website in a multisite setup, and the body of the violin is like the code base. A problem in the code base will affect all the websites, just like a broken violin body will make all of the violin’s strings useless. And if you think playing a violin with four strings is difficult, and believe me, it is, how does a violin with a hundred thousand strings sound?
[violin sound design ends]

The risks and rewards of Rich’s method for management

[Doug] So there are some risks to using WordPress Multisite to manage this infrastructure.
[Rich] If there’s a problem with one of them, there’s a problem with all of them.”
[Doug] So, why does he still use it?
[Rich] It might not sound like the greatest idea or a huge amount of time saver if you have, say, three or four websites. Yeah, all I have to do is press the Update Now button and it’s done, right?
[Doug] Just like Rich says, in WordPress there is an “update now” button that will install updates for plugins, themes and WordPress core. There are more sophisticated ways to do it, but this is how the vast majority of WordPress users do.
[Rich] So it might be 10 clicks across four sites. But when you start having hundreds or thousands or tens of thousands of websites, those clicks add up.
[Doug] My calculator tells me that 3 clicks per website equals three hundred thousand clicks for Rich.
[Doug] So the upside is clear: without a single code base like this, Rich’s job — keeping hundred’s of thousands of sites updated — would be impossible. Rich has a lot of faith in the WordPress platform and the community that supports it.
[Rich] WordPress core is remarkably resilient to getting things wrong.
[Doug] WordPress is an open source software project. What that means is that thousands of developers  all over the world contribute to improving it. They might find a small issue and report it in the online forums at wordpress.org. And when the next version of WordPress comes out, developers all over the world have contributed to a solution. This community approach to software development means that many skilled people are working together to make sure that WordPress updates cause as few issues as possible.
[Rich] Which is one of the reasons we go with WordPress because it is good. It’s really good software.
[Doug] So, Rich’s job depends not just on his own skills, but the input of this entire community. And up until this point, that strategy had been working. But soon he’d run into the most serious issue he’d ever dealt with in his professional life.
[music]

The crisis

[Doug] Let’s go back in time to July of 2015. All of Rich’s 100 thousand websites were humming along just fine. And just like he had many times before, Rich received a notification that WordPress had released an update, version 4.2.3.
[Rich] “It was a Wednesday afternoon, so I waited until close of play, updated WordPress on the staging environment. My team and I checked it all through. Everything looked fine, so we put it into production.
[Doug] Because actually looking at all the sites would take too long, Rich has a list of a smaller sample of sites that he tests whenever he needs to make updates to the code base.
[Rich] It took maybe seven minutes for the first ticket to come through! And it was “so, my whole site’s broken, nothing works…”
[Doug] In case you aren’t familiar, a “ticket” is a support ticket. That’s a request from one of Rich’s users for help with a problem with a website in the system. In less than an hour, Rich received several hundred support tickets. His worst nightmare was happening. They said…
[Rich] If you try to go here, all of the accordions and the tabs don’t work, if you click on links, they all just reload the page. I thought “oh that’s just one isolated incident.” Yeah, it wasn’t one isolated incident, it was hundreds of people sending tickets saying “everything is broken, every link is down…”
[Doug] It’s important to note here that when the users complained that “everything was broken” that the websites were still actually loading and working.
[Rich] People’s websites did still work, but it had the appearance that things weren’t working.
[Doug] So, Rich needed to spend some time digging in to see what “Everything’s broken” really meant here. And he found that across many of UBC’s sites, content that thousands of users had created was suddenly appearing as a gobbledy gook of code-like text. All the tidy charts, colors, and data that the users had spent their valuable time creating over the last several years were messed up. Even though the sites weren’t technically broken, what mattered most to Rich was this:
[Rich] I have tens of thousands of people who rely on these websites working, day in, day out. And they stopped working.
[Doug] It was time to figure out what went wrong, and fix it as quickly as possible.
[music]

What happened?

[Doug] There are a couple important questions to answer before we go on. First, why did this WordPress update have such a drastic effect on UBC’s websites? And if this update was causing so much trouble, why did Rich want to install it at all? First thing’s first, let’s look at what caused the problem. It turns out that almost every issue involved the use of shortcodes.
[Rich] Shortcodes are ways in which you can write a small amount of, in essence, code, which is interpreted by servers and outputs something else.
[Doug] When you’re editing content in WordPress, a shortcode looks just like a bit of normal text inside of square brackets. But when you view the website, you see something else entirely, like a table of data or an audio player, on the page where you put the shortcode.
[Rich] Shortcodes themselves give users huge amounts of capability, so many different options, but the problem is, if those shortcodes break, it’s very difficult for them to work out how to fix them.
[Doug] The reason it’s hard for a user to fix them is that the user can see the shortcode, and can view the resulting item that’s displayed on the website, but can’t see what happens in between.
[Rich] Because really, it’s a bit like magic, they type in some gobble-de-gook code, they press publish, and then all of a sudden, on the front end of their site, it translates into something else entirely.
[Doug] What’s happening there is that WordPress takes the content, checks it for shortcodes, and processes any that it finds according to rules defined in code. Those rules are what turn a shortcode for a table into the HTML that actually represents a table. So then WordPress injects the translated content in with the rest of the content, and then displays it for the user on the front-end of the site. Put in the shortcode, and poof! An intricate object displays on your website, like a rabbit pulled out of a hat. And magic in the hands of a muggle?
[Rich] …magic is never good…
[Doug] …Because shortcodes put the magic of code in the hands of people who may not be prepared to use it wisely.  So, why did the shortcodes cause problems? We’re getting into some in-depth technical language now, so hold on to your hats, I’ll explain in a bit.
[Rich] The issue was actually two fold: The first one was, people were using shortcodes in what are called attributes.
[Doug] Attributes are parts of an HTML element that provide extra information about the element, and generally, an attribute should contain only simple text. But, in this case, users were putting shortcodes inside of those attributes. On top of that, there was another issue.
[Rich] A second way that was actually more prominent and more problematic was “nested shortcodes.” So, a shortcode within a shortcode.
[Doug] The technical details are important here, but in case you don’t follow them, what’s you need to know is that it wasn’t just the shortcodes themselves, but also the ways people were using them that was the problem—like magic on top of magic. Now before we tell the rest of Richard’s story, we need understand what makes the shortcodes work in the first place. We need to talk about the WordPress Shortcodes API.
[music]

The WordPress Shortcodes API explained

[Doug] All right Rich, what’s an API?
[Rich] It’s the way in which one piece of software allows another piece of software to talk to it.
[Doug] In this case, the Shortcodes API is a way for shortcodes to talk to WordPress, and a set of rules for how it should be done. When a shortcode appears on a page, it’s like a signal to the WordPress Shortcodes API, which then talks to WordPress to turn them into what the user is supposed to see.
[music stops suddenly]
[Doug] Now, some advanced listeners are going think this is a quirky use of the term API, since usually an API lets two different systems talk, like Stripe.com and the Easy Digital Downloads plugin that we learned about back in Episode one of this podcast. In addition to the shortcodes API, there’s also the Theme API, and the Plugins API. These are just subsystems of WordPress itself. That’s the terminology that WordPress uses, so that’s how we’ll describe it, too.
[music fades back in]
[Doug] Now let me play what Rich said again.
[Rich] It’s the way in which one piece of software allows another piece of software to talk to it.
[Doug] The key word there was “allows.”
[Rich] The change to the API for shortcodes meant that a lot of the different ways that people were using the shortcodes was no longer supported.
[Doug] What happened in this new version of WordPress is that the rules had changed. The Shortcodes API no longer allowed users to do the things Rich described before. It’s like your violin is sounding great, and then all of a sudden a string breaks.
[snapping violin string sound design]
[Doug] But Rich’s violin had hundreds of broken strings.
[music]

Why did the API change?

[Doug] So the Shortcodes API stopped working the way it used to work. Why is that? Why would WordPress make such a sweeping change when so many users were depending on it working the way worked before? When this kind of change happens, it’s called a break in backwards compatibility. It’s a big deal in the software community, and coming up next, I’ll turn the mic over to my co-producer Catherine Bridge to explain it all.
[music]

Catherine Explains: Backwards Compatibility

[Catherine] Backwards compatibility is a mouthful, and we’re going to be saying it a lot, so we want to make sure you know what we’re talking about. Backwards compatibility is the ability for hardware or software systems to interface with each other, regardless of their current version. If an update to one system interferes with another system, then the two systems are not “backwards compatible.”
[Catherine] The two systems could be a gaming console from 2016 and a video game from 2003, or even your car’s ability to play CDs.
[Catherine] Apple is known for pushing technology to make shiny new products and software. They can introduce new technology at a rapid pace because they are less concerned with Backwards Compatibility. Their most recent iphone release is a perfect example. Their new phones that lack a headphone jack will not be backwards compatible with your current headphones unless you purchase the adaptor.
[Catherine] WordPress is known for going to great lengths to keep their software backwards compatible. But more than that, it’s a sacred, guiding principle to this community.  This means that they have to move slowly and delicately with their code and feature advances, so that they don’t cause breaking changes for some users. That’s what makes this story with UBC so unique and fascinating. What could motivate the WordPress team to consciously allow a breaking change to enter their code, when they work so hard to keep that from happening, most of the time? Stay tuned to learn about one of the few things that would motivate the WordPress core team to break backwards compatibility.
[Doug] Thank you Catherine. Stay tuned after the break to hear the very good reason that WordPress broke backwards compatibility. And now a brief word from our sponsor.

A message from Rocket Lift

Do you provide a lot of information through your website? Is it a challenge for your customers to find what they’re looking for?
At Rocket Lift, we create custom search experiences that make a lot of information easy to sift through. Improved search is one of the best ways to serve your customers.
Create custom search controls, built especially for your data. Suggest matching search terms while users type, and correct spelling typos. Deliver highly relevant results, instantly, in a smartphone-friendly layout, that users can sort and filter. Track what people are searching for, to make sure you deliver it.
You can even add an API to make results available to other platforms like your mobile apps, or third-party services.
There are so many possibilities. They all add up to making your customers lives easier, and building their trust in your knowledge, which increase the chances they’ll turn to you when they’re in the market for your products or services.
Learn more by visiting rocketlift.com/customsearch. That’s Rocket, like a spaceship, Lift, like it’s a heavy spaceship, dot com. slash custom search. rocketlift.com/customsearch
Learn more by visiting rocketlift.com. That’s Rocket, like a spaceship, Lift, like it’s a heavy spaceship, dot com. rocketlift.com

Security trumps everything…

[Doug] Welcome back. When we left off, Rich’s team were flooded with hundreds of reports about broken pages on University of British Columbia websites. It was caused by a WordPress update that changed how shortcodes worked, an update that Rich had just installed. The question we’ve been inching towards answering is this: Why did this WordPress update break backwards compatibility? Rich soon knew exactly why.
[Rich] Security trumps everything. A website, if it isn’t secure, then it isn’t really… it’s nothing.
[Doug] The WordPress shortcodes API had been changed because of a security vulnerability. When backwards compatibility is so important, security is one of very few reasons why WordPress would introduce a “breaking change” like this.
[Rich] You wouldn’t normally expect any WordPress release to break anything because of backwards compatibility, which is what we’ve been talking about.
[Doug] In this case, it was necessary, but even so, a lot of people were upset. A post about this version of WordPress on make.wordpress.org, which is essentially the official WordPress development blog, generated some very angry comments. I’ll read a couple: “you broke half the internet without so much as a “howdy do” and this one: “…aaaaaand half my client’s sites are broken.” But that last one added a smiley face emoji, just so you know. It turns out that many WP users and developers were caught off guard by this change, and felt the WordPress team could have told the community the shortcodes would be affected. After all that, security is the answer to the riddle of why all of this happened. But, that answer raises more questions—what exactly was the security vulnerability, and what was done to prevent the breaking change? We’re about to talk to someone who can answer those questions.
[music]

John James Jacoby — Perspective

[Doug] To tell you more about how the development of this particular WordPress update worked, we talked to John James Jacoby. He’s the lead developer of BuddyPress, the WordPress social networking toolkit, and a prolific WordPress core contributor.
[greetings tape]
[Doug] I started telling John about Rich’s story, but he didn’t need much context.
[John] Oh, I remember that release very vividly, because there was a lengthy discussion about the shortcode change.
[Doug] John didn’t work directly on this particular change to WordPress, but he participated  in discussions with the people who did.
[John] When the shortcode bug was identified, it became clear that the way that shortcodes allowed other shortcodes to include other shortcodes in their attributes was going to be a really hard problem to solve.
[Doug] It turns out that a lot of the WordPress security team put serious thought and effort into preventing the problems that Richard experienced. As an example, John talked about one of the major problems that WordPress 4.2.3 was trying to solve, nesting shortcodes inside of other shortcodes.
[John] There were tens of people that went through and tried to prevent that breakage from happening.
[Doug] After lots of testing, they found that…
[John] …the fact that shortcodes work this way is the problem, and that the best thing to do for our users is to prevent this type of thing from being possible.
[Doug] The results of the WordPress team’s work on this issue was clear: the shortcodes API had to change, and there was no way to avoid it. And he agreed with Richard about security:
[John] In general, security will trump any other issue all of the time.
[Doug] But, what was the vulnerability? It’s hard to explain any of this without getting tripped up with loads of technical language, but here’s the gist of it.
[John] “Developers would build a shortcode that could have a shortcode inside of it.”
[Doug] This could be any developer who makes use of shortcodes in plugins or themes that they write. But there wasn’t a strict limit to how much of this could happen. So, you could have a situation where you’d have…
[John] “One shortcode for a table, another shortcode for a column, another shortcode for a row, another shortcode that would have attributes in it in quotes that would start another shortcode that would process more things.”
[Doug] And websites would run into errors trying to process all those shortcodes, especially if there was an error in how the user wrote them.
[John] “A broken shortcode syntax, a missing bracket or a missing quote would cause problems for other shortcodes, and it would cause breakage.”
[Doug]. And the security risk comes in when a user with bad intentions can turn that breakage to their advantage. They could…
[John] “…exploit that WordPress installation through comments, or through posting weird shortcodes that cause breakage.”
[Doug] So, a hacker could post a shortcode to a website’s comment area, and as WordPress breaks while trying to process it, a malicious javascript command could run in a victim’s browser. This could put the personal information of website users at risk, and that risk was unacceptable to the WordPress Security team.
[music]

Back to the story: solving the crisis

[Doug] To keep millions of websites secure from malicious hackers who could exploit broken shortcodes, the WordPress team had to break backwards compatibility, even though it caused serious headaches for Rich. Where we left off in his story, hundreds of sites in Rich’s care were broken, complaints were flooding in, and the pressure was on. The first thing he did was to revert his systems back to the previous version of WordPress.
[Rich] And then it was a process to see if there was anything we could do about it automatically. And then it was realizing that we couldn’t do anything about it automatically.
[Doug] Rich and his team hoped they might be able to engineer some kind of fix from the top down. But, no dice. They found that each user would have to fix his or her own content on their own.
[Rich] So then we had to find out the scale of it. We had to get in touch with all the people who were affected, which was around 450 people to explain the situation.
[Doug] They found that each user would have to fix his or her own content on their own. Explaining a complex situation to a large and diverse group of people is no easy task. Unfortunately, Rich didn’t have an episode of Hard Refresh he could ask people to listen to.
[Rich] Then we had to say “ok, here’s the deadline for when you have to update your sites.” We’d give them a list of all of their sites and all of the pages where the problems occurred. And we also wrote loads of documentation and gave presentations to the faculty support groups…this all happened in the space of a week and a half.
[Doug] While all of this communication was happening, Rich continued to work on the technical side of the problem. In order to preserve some functionality that his users depended on, but also shield them from the security vulnerability that had been discovered, Rich had to do something he’d never done before.
[Rich] We technically were running a hacked version of WordPress. “Hacked” as in I implemented a version of the security patch separately, in a slightly different way, so it was secure, but without running the actual WordPress update. And that was the first time in my professional career that I had A) hacked core and B) not been up to date within a matter of hours.
[Doug] It was a delicate surgery, meant to relieve symptoms and cure the disease at the same time.
[Rich] We looked at the actual patch that WordPress had made, and we reverse engineered the patch so that it wouldn’t break, but still remained about as secure as it could possibly get it compared to the original patch.
[Doug] To be clear: This hacked version of WordPress that Rich was now running on thousands of sites wasn’t totally secure.
[Rich] That is correct. Which is a horrible situation, It was one of thoses things that was, ultimately speaking, above my pay grade.
[Doug] Next up, Executive Producer and Rocket Lift President Matthew Eppelsheimer adds another perspective on the difficulties of managing this kind of crisis.

Conversation with Matthew: Managing Risk

[Matthew] Can I break in here?
[Doug] Oh, hello, Hard Refresh executive producer Matthew Eppelsheimer! What brings you here?
[Matthew]

I just want to point out that Richard Tape is a very smart guy
managing thousands of websites
hacking WordPress core
communicating delicate technical information with nervous users.
It’s an incredible balancing act he’s doing
and yet, when he says this is above his paygrade, he’s…

[Doug] — Just being modest?
[Matthew]

Well maybe a little
but the point is this challenge was enormous
— and so in a way he’s right, it was above his paygrade — it was above just about everyone’s.
Dozens of WordPress core developers couldn’t prevent this
The change was necessary
but guaranteed that people would experience challenges like Rich’s
people like him got the worst of it

[Doug] Got it. Thanks for stopping by, Matthew! Great to see you.
[Matthew] Totes. You too.

Anyhoo… Back to Rich’s Recommendation

 

[Doug] You guys, I had no idea Matthew was going to do that. All right, anyway. At this point in the story, Rich had done all he could. He had to present a recommendation to his boss, but unfortunately, it didn’t include any easy fixes, only hard choices. But they opted to go with Rich’s plan: They would run Rich’s hacked version of WordPress on the UBC system while users had a set amount of time to fix their own content. After that, Rich would implement the full version of the update, whether the users were ready for it or not. It would take a lot of work to get it all done.
[Rich] Which is why, ideally, this would have taken two months to fix, which would have given our faculty support units time to make all these changes and properly test this stuff. But we had to accelerate the timeline on it to get it done.
[Doug] Implementing his own security patch like this, when the stakes were so high, was a gutsy move. But, far from feeling invincible, Rich was nervous. And frustrated, but we’ll get to that soon.
[Doug] After about three weeks, Rich, his team, and all the hundreds of website administrators at University of British Columbia had done an amazing amount work.
[Rich] We basically got 600 or so sites to change and 7000 or so individual pages changed in about two weeks.

Richard’s Rant: Why should we care?

[Doug] When he had fully updated WordPress, and closed the security vulnerability, Rich was relieved. But he had a lot on his mind. Enough to write a blog post about his experience.
[Rich] This post is not normal fare for this blog. Normal programming will return shortly. I had to create a new category called rants, so that says a lot I think. I’m frustrated. I’m so conflicted with the shortcodes API change. I understand it’s a security fix, and that’s the trump card. I’m fully cognizant of the fact that this change hasn’t affected a huge number of sites, and that’s a really great thing. And I’m also cognizant of the fact that the way the API was being used wasn’t the envisaged way, using shortcodes in attributes, and that, to a lot of very experienced people, is silly.
[Doug] And that was just the first paragraph. I asked him what troubled him the most about the situation.
[Rich] For me the frustration was the lack of backwards compatibility in this change. And the big reason that WordPress now powers 26% of the web is because it’s always maintained, to the best of its ability, backwards compatibility. And I saw this as a really big breaking change.
[Doug] Rich isn’t just complaining to complain here, he’s trying to come to terms with a crisis of faith. And that’s because Rich is a true believer in the ideals of WordPress’ open source project. He says WordPress…
[Rich] …it has literally changed my life, for the better.
[Doug] Rich is from Manchester, England, and it was a job in WordPress development that brought him to Vancouver.
[Rich] There were two people who initially sort of made WordPress. Matt Mullenweg, and Mike Little. Mike is from my hometown. I used to live down the road from him. Mike wasn’t the person who got me into the WordPress world, but he was the guy who made me realize how amazing this thing is.
[Doug] Rich felt better when Mark Jaquith, one of the lead developers who worked on this release, commented on Rich’s blog post. Here’s a bit of it: “The WordPress team (and I personally) erred seven years ago by not being conservative in the shortcode use cases that we allowed to work, by not saying “these ways are supported, but THESE ways are not”. There were earlier versions of the security fix that would have broken even more plugins, and a lot of work was done to minimize the impact. We should have communicated the changes and shared information the supported/unsupported use cases before there was pushback by plugin devs and plugin users.
[Rich] Mark is very good with writing words. Mark just laid it all out.
[Doug] In the end, Rich felt the WordPress Core team made the right decision. The more he learned about all they did to avoid the breakage that caused Rich’s troubles, the more he was able to get past his personal frustration.
[Rich] I know a lot of time and effort and person hours went into looking at this thing and trying to work out a way that they could fix it without breaking it. Sometimes you have to break something.
[Doug] So Richard and everyone at UBC learned a lesson, but did the WordPress community learn one as well?
[Rich] I think for every release that happens for WP the people behind it improve and get better, I don’t know how because they’re all amazing. But, they get better every year. Honestly, it’s ridiculous. They’re smart people. I think they learn lessons every time they press the commit button. And that’s why WordPress gets better and better.
[Doug] So, lessons were learned, and by many different people. Rich learned how to manage a crisis of massive proportions, and gained a new respect for the WordPress community. And the WordPress community learned an important lesson too—that communicating as early and clearly as possible about breaking changes like this will help users and developers both to adjust as painlessly as possible. Finally, no matter how challenging this situation was from Rich’s perspective, there’s no point in working so hard to maintain any website, let alone a hundred thousand of them, if they’re not secure.

Closing Credits

[Catherine] Recording, editing, mixing, and sound design by Douglas Detrick. Production  by Douglas and me, Catherine Bridge. Matthew Eppelsheimer is our Executive Producer. Our amazing violinist was Casey Bo-ZEHL. Be sure to subscribe on your podcast platform of choice, and please write us a review on itunes. Your review helps others find us. Head on over to hardrefresh.audio to learn more about Richard or read his rant post to the WordPress core team.
If there was something in this episode that you have questions about or didn’t understand, we want to hear about it. We’re making this podcast for you! Leave us a comment at hardrefresh.audio, send us an email, or find us on facebook or twitter.
Coming up next time, on episode three of Hard Refresh: What happened when a web development team was really excited to launch an updated version of their own website, and it went wrong in just about the worst way possible.
[Matt Pearson] So I was watching this torrent of text just roll across my screen, thinking about rocketlift.com because we’re working on rocketlift.com, and my eyes caught emyth.com in there, and my brain went, “Oh, that’s wrong. That’s not the site that we’re working on right now! Something is terribly amiss.”
[Catherine] Want to be on the show? If you, or someone you know, has run into a serious problem while working on the web, we’d like to hear about it. We’re looking for stories where internet technology was part of the problem, and human creativity was vital to the solution. Send us a message at hardrefresh.audio.
Don’t forget, if your website provides a lot of unique information, check out rocketlift.com/customsearch to learn how Rocket Lift can help you better serve your customers.
[Doug] That’s episode two. Thanks for listening. Now, perhaps it’s time treat yourself to a hard refreshment?