Behind the Screens Halloween Special

0:00 0:00

It doesn't matter who you are or how long you've working in the industry, you are a human, and that means you have tasty braaaaaaaaaaaiiiiiiiiiiinnns! Ugh, excuse me. I mean, you've probably made a mistake. Members of the community share their scariest moments in the industry, from deleting large sections of a web server, to building Drupal sites in Flash.

Guests include:

Thom Toogood
Greg Anderson
Mike Anello
Jen Witkowski
Jim Birch
Kris Vanderwater
Rick Manelius
Angie Byron
Chris Albrecht

Transcript

Speaker 1:: Front end developer for the first time in my career.
Speaker 2:: User experience design.
Speaker 3:: I'm a backend developer. I mainly work with Drupal.
Chris Albrecht:: Welcome to another episode of Behind the Screens where we go beyond the code and into the community. I'm your host Chris Albrecht. Let's dive in.; Behind the Screens is made possible by Lullabot. Lullabot is a strategy, design, and development company, where we turn great content into captivating user experiences. Find us on twitter @Lullabot or in your browser at lullabot.com. We'd love to meet you.; Welcome to a very spooky edition of Behind the Screens. In this episode we are talking with people from the community about their scariest moments and the things that keep them up at night. Hold on to your keyboards.; Thom Toogood.
Thom Toogood:: It was a little while ago. I was working for a big utilities company. They were running their AGM at the time, and we were streaming live, this event on the website. For some reason I actually didn't know it was going on, and there was an update available, and I thought, "Hey, you know, perfect time to run some updates."; I triggered the updates on the site. Running update PHP, this was Drupal 6, a little while ago, and luckily it all worked, and no one actually found out. It was probably the worst time to run your updates, in hindsight.
Chris Albrecht:: Yeah, soon as you hit that button and you realize what's happening, and you go, "Oh crap, please don't crash."
Thom Toogood:: Yeah, it's a big company and it would have probably made the news. I would have probably got in a bit of trouble, but luckily the update worked.
Chris Albrecht:: Greg Anderson from Pantheon
Greg Anderson:: I had a really horrible experience once, and our IT department helped me with this experience. Before I started working at Pantheon, I used a Linux box. I was always on a Linux laptop, and really liked that experience, but every now and then for my work, I had to switch over to Windows. Well I had to dual boot Windows machine. At one point in time I was dual booted into Windows and even though I use Linux a lot more than Windows, I sort of thought I knew how to do Windows networking. By God, I was in there, it was Windows XP era, trying fiddle around with that little thing and it wasn't doing its thing. One of the IT guys happened to be walking past my office door, and I said, "Hey, could you just help me fix my internet?" He's like, "Sure, I can do that, just let me take your laptop back to my cube."; I've got some words of advice to your listeners; never let IT just take your laptop back to their cube. Especially if it's a dual boot machine booted into Windows.; When he came back he's like, "I got your network fixed and you are going to love me so much, because your computer is going to run way faster than it used to, because I defragged your hard-drive."; I said, "You did not defrag my hard-drive." And he said, "Yes I did defrag your hard drive." I'm like, "What about the Linux partition?" He's like, "What Linux partition?" "The Linux you just erased." He said, "No, it's not going to erase the Linux partition, it's only going to erase the Windows partition." He was wrong
Chris Albrecht:: Oh no.
Greg Anderson:: Yes, this was absolutely horrible for me, because I have really good practices, and I lost not one bit of work material. All of that was backed up, but this was at a time when my son was a toddle, he was one or two years old. I had some spare time projects that I was working one, and my back-up mechanism is, every time I finish something I commit it and push it up to the repository. So my laptop, I can throw it away any time. IT can take my laptop and defrag my hard-drive, I don't care. So the problem with this, for the last year, my method of working is, "Oh my son fell asleep I'm gonna get a little work done. Oh my son just work up, I'm gonna close the laptop lid, set the computer aside, do what I need to do with him." So when I got my computer back and I went to clone the repository, I discovered I had not made a commit in an entire year. So I lost an entire year of work on my personal project and it was so emotionally devastating that I never worked on it again.
Chris Albrecht:: Mike Anello from DrupalEasy
Mike Anello:: The things that stick in my mind are, they're almost like when you have a nightmare, and you wake up and everything is okay? I would suddenly think about, oh my gosh, did I do something really wrong, an hour, a day ago? Have that cold sweat. Did I secure that thing? Did I accidentally copy a database up to production? Did I do something horrible and have that sweat and worry until I had five minutes to go and check it, and say, "oh okay, I'm covered."
Chris Albrecht:: Everything's all right.
Mike Anello:: For me that's horror, and luckily it hasn't panned out to be actual horror, yet. For me that's pretty damn scary stuff.
Chris Albrecht:: Did I accidentally transpose the aliases in that trust command?
Mike Anello:: Yeah, exactly. Another one is stuff where you get an e-mail from a client saying, "our sites down, what's going on?" As the developer I'm like, "I'm not hosting your site so ..." but did I do something to cause that? Then I go check everything. Actually this just happened to me a couple of days ago where I get panicked e-mails from the client, and it turns out that it was a mis-configuration of their internal network where they couldn't access their own website.
Chris Albrecht:: Oh, gosh.
Mike Anello:: So there I am panicking, trying to figure it out. I haven't push up in a week or so. I haven't done anything else. So I try to help them out, but in the end I was like, "Okay, I'm cool, I'm covered."
Chris Albrecht:: That is slightly terrifying.
Mike Anello:: It's like a nightmare you wake up from and everything's cool.
Chris Albrecht:: Jen Witkowski from Lullabot.
Jen Witkowski:: This actually happened recently, [inaudible 00:06:33] big tax blunder. It actually happened with kind of a small group of stakeholders. We were on an on-site recently and we were doing ... When we're on an on-site we do a bunch of exercises with stakeholders to understand the parties goals and what type of visual styles they're drawn to. One of the exercises that we're doing is something called the five .. is it five seconds? It's a gut-check basically. I can't remember if it's five seconds or if it's 20 seconds. It might be 20 second gut-check, where we show them a website. The stakeholders have 20 seconds to rate it, if they like, or if they don't like it, then move on to the next one. In order to do this you have to set up PowerPoints, and each slide has a website on it with a number in the corner. The numbers are labeled 1, 2, 3, they go [inaudible 00:07:37] the PowerPoint keeps advancing. Those numbers kind of align to the worksheets that the stakeholders have. I had everything all set and at the last second I decided to re-arrange some slides-
Chris Albrecht:: Uh Oh.
Jen Witkowski:: -Not realizing that I'm also re-arranging the numbers, and I also duplicated slides and changed out some of the images and not changing out the numbers. After explaining the whole process to the stakeholders and getting them all psyched up, we started the PowerPoint and it started on number 2.; Everybody is like, "Do we start with number two or do we start with number one?"; I was like, "Oh crap." I immediately knew what I did.; So we were just like, "You know what, it's first thing in the morning, we're gonna keep you on your toes, just take a look at the number in the corner and write the number down, and cross out the numbers that are already down there."; It worked out, it worked out fine. I think we only had one worksheet that we turned in that we weren't exactly sure what was their meaning, because there numbers scribbled out, then there was arrows pointing to other numbers.; We were like, "Yeah." So learnt my lesson.
Chris Albrecht:: Jim Birch of Xeno Media
Jim Birch:: You know I know people have deleted servers before, and published pages they haven't. I'd say that my biggest horror story was for a company that made food products. This was my first Drupal theme, and I got a design from a mock up, and it had a beautiful picture of the food on the front page. It was a four page website, it was built into their multi-site, it didn't really need to be Drupal, but this was much to late in life to have a full-page flash movie play, with music on, in my first Drupal site. I would say that was my biggest horror, and it's still on there.
Chris Albrecht:: And it gets away.
Jim Birch:: It continues to haunt me today.
Chris Albrecht:: Kris Vanderwater of Acquia.
Kris :: When I joined Acquia, I actually joined in the marketing division, and that was a lot of fun, but for various reasons I ended up moving over to PS. The first big project I really helped out with in PS was an early build on the Drupal 7 version of Lightning, which is Acquia's install profile for, kind of encapsulates a lot of our best practices of what modules we think you should be using, and what basic config you should be using. Very early in its life-cycle, it wasn't really ready for prime-time development, and we had decided that we were going to go ahead and use it on a project. Trying to get revision handling of layout in Drupal 7 is a trick, even today, so doing this three years ago was quite the trick. There was a lot of re-arranging exactly how panels and workbench moderation all integrated together or rather [panelizer 00:11:05]. Yeah to call it a horror story might actually be a bit of an understatement, I'm not sure I can get more detailed than that though.
Chris Albrecht:: Rick Manelius from DRUD Tech.
Rick Manelius:: Let me tell you a story that gave me literally, probably my first gray hairs. A company that I worked with which was selling online based memberships for personal development, and we had a good series of sales over about six months, and our credit card company called us and said, "Hey, we noticed you're not PCI compliant." I didn't know what that word even meant. I was just like, "I'm a developer, I just ... " If the credit cards are going through it means we're good.; The credit card company says, "Oh no, you've actually crossed a volume over six figures of sales, by which you have to become compliant, and if you're not complain in 30 days, you're going to have to refund that six figures worth of transactions, and you will no longer be able to accept credit cards."; And our entire business model is built around this. Kind of that whole imposter syndrome, vulnerability, I had a lot of trust with my employer at the time to actually go to them and say hey this is a problem, and not try to just fix it in the background. He immediately freaked out as you can imagine. I had to do the PCI compliance crash course in about 30 days, and I wouldn't say that we were compliant out of the box, or out of the gates, but we eventually over about a six month time period achieved in maintaining that compliance. That really sparked my interest to sort of help others [inaudible 00:12:42] no go through that same moment of terror of, just because you can accept credit cards doesn't mean you have the authority to do so.; Helping educate the community through the white-paper, and being on a couple webinars, like the commerce guys who are incredibly supportive, DrupalEasy and Mike Anello and a bunch of other things. That was kind of a big splash for me too, because that was one of those things where I didn't feel I was smart enough to be a thought leader on that topic, but nobody in the community stood up and was that, so I kind of just did it. Thank god for Greg [inaudible 00:13:15] and standing there and checking my work, because he is head of the security team, leading the security team. I needed someone to prove that I knew what I was talking about. That was transformative event, so I took a very horrifying, almost near panic attack experience, and converted into something very good for the community. I hope others have, when they've hit those areas, have been able to use it for good as well.
Chris Albrecht:: Angie Byron of Acquia.
Angie Byron:: I used to work a Lullabot, a little before your time. While I was working at Lullabot, this was my first big project with Lullabot, and it was a data-migration of something like 200 thousand records from Vignette to Drupal 4.7, I think, really old, like this is a long time ago. I did the thing that you did back then, which is you spend tons and tons of time writing this crappy mischief of SQL, PHP, and Bash, and whatever you gotta do to get a script that will get things moved from point A to point B. We didn't have my great API or you kids get of my lawn, it was this macgyvered rubber-banded thing. Stayed up til four in the morning or some ridiculous time, ran the script, ran forever, imported all 200 thousand records, checked it all, everything worked great. I was like, "Awesome!", and went to bed.; Got up the next morning, Oh, there are 20 more thousand records we forgot to tell you about. So now I'm like oh my god. I'm trying to find this script that I just totally hacked together thinking it would only ever have to run once, and suddenly it would need to run again a second time. I'm going back through it and I'm making sure, and again another four a.m. night and this was on a weekend. Got the script running locally, tested it, we're all fine, ran it on production, 20 thousand records, checked a couple, seemed to make over fine, great, go to bed. Get a call, Matt Westgate, six a.m. on a Sunday, he was like, "can you get on the internet? There is a problem."; Long story short, the sequences table is not your friend, and what I had done is I had imported those extra 20 thousand records and everything had moved properly, but back in those days you had a table called sequences that would keep the last node id, that you had put, and I forgot to update the sequences with the latest node id. What was happening is node ids were being re-used so content that said it was one thing was another thing and it was a whole ...; So I, of course, thought I was going to get fired. I thought I would maybe get sued also, because that was a horrible thing. So I'm like totally distraught.
Chris Albrecht:: So this was all done and it went live on production, it's out there for the world.
Angie Byron:: Oh yeah, it's out there, like the companies lawyers are the people who discovered this problem.
Chris Albrecht:: Oh boy.
Angie Byron:: It's not a fun time at all. I fixed it in four seconds. Once I knew what it was, it was a really easy fix, because it was just a dumb, I think I had done that at the tail end and it didn't make it's way into the script, because I would never have to run that script again, I thought.; So anyway, I'm panicking and thinking, convinced my life is over. Matt calls me after the client got calmed down and he's like, "So we're going to get you a massage."; I just started crying, because I was like, "oh my god" because I thought for sure, it was the total opposite of what I thought was going to happen. So they got me a massage, they got it booked for me in downtown Montreal, and then said, "And from now on you are the data migration person because I know you will never do that again"
Chris Albrecht:: Chris Albrecht from Lullabot. So this one goes all the way back to my start in tech. I'd just taken a job where I thought I would be working on the website. It was mid-2000's early-2000's, and the website was a mis-mash of PHP code largely written by PERL developers. Like a big homespun CMS that handled registration for things, payments and checkout, as well as an organizational tool to group people together and create teams, and the brochure front-facing website of the company as well. When I came in, I found out that the head of their tech department was on his way out, and he handled everything from the server configurations, the DevOps, the deployments, and all of that sort of stuff, and that was a world I was completely unfamiliar with. I knew HTML, CSS, a little bit of JavaScript at that time, this was back even before Jquery, I think, came around. I ended up getting in way over my head. They wanted me to take over his job and I told them, "Look I can't tell you the difference between Unix and Linux right now, other than they both end in n i x."; And they're like, "Ah, don't worry about it, you'll be fine. You'll get trained up on all those things, and everything will be okay"; Sure whatever you say. So they train me up as best they could, and I have root access to the server and I'm moving things around. One of the tasks I needed to do was to scan through a series of parallel web directories and remove any htaccess files that might have been added to them. So I'm like, "Okay, I could do this one by one and be here all day, or I can craft a nifty little script that will go through each one and wipe them out."; So I started putting script together, "-rm .htaccess" ...something something and there ended up being a space and a star or two in there maybe a slash. So I tested the command in pieces then ran it all together, and figured it should be real quick to go in and wipe out the file. It's running, and it's still running, and it's been running way to long. So I canceled out of the operation, I go back in and check and our web server is missing any directory that begins with the letter A through H. That's not usually good for a web server.; I ended up having to call my other coworker, who really should have been, he knew way more about this stuff than I did, and it was 9 pm eastern time for him. He had to initiate a backup restore from our host, and we had to go in to manually start piecing things back together. Luckily that was all there and I don't think we really ended up losing anything except what might have been done during that day. We had to explain to a lot of people why their FTP space might be missing a few files if they had worked on it earlier. In the end it's one of those situations where I'm never going to make that mistake again, and I haven't since.; That's all for this episode. Don't forget to backup your databases and commit your code.