Scary Web Development Stories to Tell in the Dark

Matt Kleve: For October 20th, 2022. It's the Lullabot podcast.

[Intro music]

Matt Kleve: Hey everybody, it's the Lullabot podcast, episode 260. I'm Matt Kleve, a senior developer at Lullabot, and today we're talking about spooky season and telling ghost stories. As we approach the end of October, it turns into the season of witches and werewolves and goblins and cute pumpkins and trick or treating and well, there's scary stuff too. So today we've assembled a group of lullabot's to tell scary stories from the histories of their careers in web development. It is important to point out that we're not trying to disparage or trash any of the the previous employers or clients or vendors that we're talking about in these stories, but just kind of acknowledging that sometimes computers are hard and bad things happen and people end up reacting to that. So first up, telling a scary story. We have senior developer Matt Oliveira from Ontario, Canada. Hey, Matt, what's your story?

Matt Oliveira: Hey, Matt. Yeah, this is from a while back it's before I joined Lullabot, I was at a smaller agency and we were doing work on a project for a political think tank. And being a political think tank, they needed to raise funds, of course. They're a nonprofit. And so we did a Drupal seven site for them with CiviCRM integrated. And I haven't worked with Civi in a while, but back then it had its rough edges. And so but anyways, we went about the project, we put it all together. Important for the story is that they had a donation page where they could take donations, and that was a CiviCRM piece. And then also important was that they needed a specific payment processor, like a Canadian payment processor called Moneris. And that didn't come out of the box with Civi. That was something that I scoured the internet for and gratefully at the time found somebody had written it. And so I downloaded it, installed it, tested it. Everything worked well with we tested with like, really low dollar donations just to see things go through. It was good. So we launched the site. Donation form was up, donations started rolling in, and we kind of moved on. It was it was one of those small agencies where you're like on a project for a month, maybe, and then you're on to the next. So that was kind of the last we heard for a bit. And then but then a couple months later, the client came back and there was some discrepancies with the donation amounts that they were getting in. They were getting actual amounts in their bank account that was much, much lower than the reported amounts in CiviCRM. And so like we had a couple examples to look at. And it turned out like after a while of looking, a pattern became super apparent where like thousand dollar plus donations were getting rounded down, so like $1,000 donation was getting rounded down to $1. So $1 only came through.

Matt Kleve: Oh oh my. So was the charge for a dollar?

Matt Oliveira: The charge was for a dollar.

Matt Kleve: Okay. So at least the the credit card transaction happened in a way that wasn't detrimental to the person trying to charge their card. But the organization didn't get their money even though their CRM said they were getting their money.

Matt Oliveira: Exactly.

Matt Kleve: Scary, right? That's a rough situation.

Matt Oliveira: Yeah. So like, obviously this is an emergency thing. I got put on it for a whole day. I did debugging, trying to chase down the code that was was causing this. And as it turned out, the so the dollar amount that was submitted in the form was a formatted string. So like for $1,000, it was like it came in as one comma, 000.00. And then somewhere deep in this payment processor code that I had mentioned, I grabbed from the internet, it was just casting that string as an integer. And when you do that in PHP. Php is going to look left to right and it's going to say, oh one that's cool. And then it's going to encounter that comma and be like, oh, I'm done. I'm not looking at that. So that's how a thousand turns into a dollar and 10,000 turns into $10. We had a couple of those, too.

Matt Kleve: Oh!

Matt Oliveira: So. Yeah, simple bug fix.

Matt Kleve: Thanks PHP.

Matt Oliveira: Yeah, thanks. It was a simple bug fix. We got it out, and then we got some interns furiously calling donors and asking them if they would please resubmit their donation. It turned out in the end.

Matt Kleve: But that's that's a rough phone call to make. But I suppose if it's somebody giving thousands of dollars, they aren't aren't they aren't upset to hear from somebody. So that's a good thing.

Matt Oliveira: No. Yeah. That level of commitment I guess there they were okay with it and everything turned out okay.

Matt Kleve: That's. Yeah, that's one that keeps you up at night. Wow.

Matt Oliveira: Yeah. For sure. So rigidly test your donation forms, everybody.

Matt Kleve: Happy Halloween.

Matt Oliveira: Happy Halloween.

[Intro music]

Matt Kleve: So up next we have Erin Schroeder. She's a senior content strategist at Lullabot. Hi, Erin.

Erin Schroeder: Hi.

Matt Kleve: I understand you have a story to tell about a death march on a project, if you will.

Erin Schroeder: A spooky story.

Matt Kleve: Yeah.

Erin Schroeder: I feel like I should have a flashlight under my face when I tell this. So there's a. We were working on a project, it was a website redesign for a a client, and it was moving platforms, right? Moving from one platform to a new platform. And we did a great deal of content work. Besides just setting up the structure and navigation of the website there, we identified opportunities for content to be consolidated, pages that were definitely missing and content that definitely needed to be created and written. We had a lot of pages that were pretty good as is and were ready to just migrate over. But because the system we were using and the system we were moving to weren't really there wasn't really an automated way to do this. So it was a lot of manual manual lifting and kind of moving over to the new home. And and I really wish I knew how it got to this point, but like maybe a day or two before launch, we realized there was a good 300 and some pages that had still not been moved over and needed to be moved over for day of launch. And the client called and was understandably upset, very, very anxious a little bit tearful that, you know, this was missed and it was and she she admitted it was her overlook. But she said, what can I do? What can I do? You know, some of these pages need to be edited. Some of them need to be fixed and, and updated a little bit to match the new style and, and align with best, you know, better web practices. So she really had all these lofty goals for this content, which as a content strategist, I love.

Erin Schroeder: But I knew that time was of the essence, so we managed to find some hours in the project to assign me to that basically round the clock for two days to move those pages with her clean them up update them a little bit, refresh them, you know, not not a great deal. We didn't have any time to do, you know, stakeholder interviews and actually rewrite content, but actually just kind of improve the look of it, improve the design and the structure of it to a way that it was it was still making that vast improvement to the new site when it when it went live. And needless to say, long story short, we got it done with some sweat on our brow and a big high five. And I still have on my on my corkboard in my office, the the thank you card that I received from that client that day for us working so great together and it was it was really rough. But we did it. We really did it. And it was a good learning experience. I would never recommend anyone get down to that wire and try to do that on their own and and get down to the 11th hour of of migrating content. But we did manage to do it. And I'm really proud of it. But it was it was a panicked 48 hours for sure, but really, really proud of the work we did and that we were able to crank it out and get a fantastic launch out on on the day it was due out. So we hit all the deadlines and then some.

Matt Kleve: All's well that ends well?

Erin Schroeder: All's well that ends well. That's correct, that's correct. And and it was good to she, you know, with any web project the web's never done. So we knew that there were more content needs that were going to come in the in the future. And so we worked together later on after launch to continue those two. But for day of launch, we had quite a few pages that were just just needed needed a home and they needed a home fast. So we just rolled up our sleeves and got to work and we made it happen. But it was it was a pretty panicked a pretty panicked task for 300 and, I don't know, 350 pages of content to be moved to their new home within, you know, 24 or 48 hour launch period. So.

Matt Kleve: Have you seen those like joke, copy and paste computer keyboards like that would be when you would want one of those, I think.

Erin Schroeder: Absolutely. Yeah. If my keyboard had nothing else but a control C and control V, that would have been just fine.

Matt Kleve: But it sounds like you ended up doing more than just, you know, manual migration. You...

Erin Schroeder: We did. Yeah, no, we did. We a lot of the content that had been written before was written, I don't know, kind of like a term paper, which happens with a lot of body text where it's just gray chunks of paragraphs. And we knew going into this project, we wanted to make sure that the content had a little bit more ease of reading. We wanted to make sure it had more subheads was easier to digest than smaller devices, you know, kind of divvied up the content, made it easier to read. So while we couldn't rewrite as much as we wanted to get done that came later, we did manage to just kind of review the content as we placed it and say, we could add a paragraph break here, we can add a subhead here, we can add bulleted lists. So as a tip for anyone listening, if you don't have the resources to rewrite your content, sometimes just fixing the way it looks can make it improved and make it easier to read. So we did that with quite a few pages and we saw big improvements. Even with just those small tweaks, we saw big improvements in the analytics after those went live. But again, rushing to do them. Not ideal, but we did them. But if you're really on a, on a stretch, just, you know, add a few paragraph breaks, add some bulleted lists, make the content a little easier on the eyes, and you will see improvements, I promise.

Matt Kleve: But you can do hundreds of pages in two days. It's possible. So what's your excuse? Get to work.

Erin Schroeder: [Laughing] Exactly. I mean, if you like having your heart race at a hundred and, you know, 70 or 65 beats a minute all the time until launch day. Live on the edge, you know, go for it. I don't recommend it. I think we and if we had not, you know, if she had not miscalculated those those pages I think we probably would have been in a much better situation and would not have had to rush and would have gotten probably a little bit more hands on stuff done. But needless to say, we did get it done. But I do not recommend that if you are not someone who loves stress. I am not someone who loves stress. I have a meditation app or three on my phone, so I would prefer that that hadn't happened. But we did it together and that was the the big thing. So it was a very good bonding experience in the end.

Matt Kleve: One thing you said there, the web never stops. Is that what you said?

Erin Schroeder: Correct. Yeah.

Matt Kleve: It reminds me of Newman from Seinfeld. The mail never stops!

Erin Schroeder: The mail never stops! The webs never over the web never stops. It's never done. It's never done. It's nice to, like, wipe your hands off and walk away and say like, oh, good, all the content's up, but you need to watch that content like a little, you know, plant and watch it grow and make sure it's watered and make sure it's cared for. You know, it's it's it's a very much a TLC process with your content. So and after we got done with this project, we continued to nurture what we had put up and make it better and continue to make more improvements. So I would encourage people to do that too. I mean, take a break after you launch a site, you earned that, you earned that rest and relaxation and that breather, but come back and keep working on your site because the web is never really done.

Matt Kleve: Happy Halloween.

Erin Schroeder: You two, Matt. Thank you.

[Intro music]

Matt Kleve: So we got David Burns, the director of support and maintenance at Lullabot. Hi, David.

David Burns: Hello, Matt.

Matt Kleve: So, David, I understand you have a story to tell me about a tricky Drupal bug.

David Burns: Yeah. This one really pushed my developer skills to the limit. This was back in Drupal five days.

Matt Kleve: That's scary enough already, right? Drupal five. Shoot.

David Burns: Right? And to make it even more challenging, this is when I was just using a regular text editor to do my development as opposed to a IDE.

Matt Kleve: Okay, yeah, that was the normal thing to do, though, right? I mean, text wrangler. And, you know, just do it.

David Burns: I believe I was using Coda back then.

Matt Kleve: Okay. That works. I mean, people use vim today, so whatever. Right? But.

David Burns: Good point. So I was working on a bug. I forget whether it was like a contrib module or a custom module. But for the life of me, I couldn't get any any of the code I was putting into the module to, to show up on the screen. So, you know, you start doing print and die and put like just a random word that you're expecting to a white screen, the page, and nothing was happening. So I'm like clearing caches, doing everything else and still nothing. So I don't know how I stumbled across it. Maybe I was looking like for the name of the file, and I went to the terminal to do it, and I did it from the root directory of the project. And in Drupal five days, there's a Drupal module inside of the modules directory. So right off of your docroot, you have modules and then Drupal. And if I open that folder, there was an entire installation, like a copy of all the files of Drupal in that Drupal modules directory.

David Burns: And I guess the way Drupal does like specificity for which module it should load for like multi-site it keeps nesting down and basically the furthest module down, it would load the contents of that file. So this Drupal folder inside the modules directory wound up pushing it out like five, five layers deep, whereas the sites all modules had it maybe like three, four layers deep. And so it was picking up all those nested Drupal installation, and it literally took me a day and a half to figure out what was going on.

Matt Kleve: It's the stuff of nightmares, man.

David Burns: Oh, do I want to give another story?

Matt Kleve: Sure.

David Burns: And you could. You could pick and choose which one you want to add. Alright. So this was back when we had the Lullabot store and it was Labor Day weekend, and Matt and Jeff were like, hey, we're going to do a 20% off sale. Go ahead and wire it up. And back then it was before Drupal Commerce. What was the name of that?

Matt Kleve: Uber cart, right?

David Burns: Uber cart! Yeah, it was fairly robust because there was a discount field that you could just be like, everything in the store gets a fixed rate discount. And while I was testing this, there was a checkbox to do like sandbox mode. So you could type in any credit card number, hit submit. And it goes to like some fake endpoint and just confirms back. Well, get everything wired up, push it out to the live site and the day the sale happens. And then at the end of the day they're like, huh, none of these credit card transactions went through. Oh, apparently I left it in sandbox mode the whole time.

Matt Kleve: Oh, man.

David Burns: And had to give a long report of all the sales that we had, along with the email addresses, and we actually hired somebody at the time to reach out to those people and let them know of my mistake. But Matt and Jeff said, anytime we have a sale on the site we're going to have you set it up, Dave. Because we know you're not going to leave it in sandbox mode ever again.

Matt Kleve: Well. Thanks, Dave.

David Burns: Yeah, thanks for having me.

[Intro music]

Matt Kleve: And so up next, we have the president of Lullabot, Brian Skowron from the Dallas, Texas area. Hey, Brian.

Brian Skowron: Hello, how are you?

Matt Kleve: I hear you, I'm great, I hear you have a story for me from your from from the from the past days where you were working in the web hosting industry.

Brian Skowron: Yes. This was me way more than a decade ago. But I there was a, there was a particular NFL team, a very prominent NFL team that was hosted by a company I used to work with. And that team's website went down at around midnight one day which caused a mad scramble for people to try to figure out why the site was down. And there was a whole lot of back and forth heated you know, phone calls and emails and such. And it turns out that they had let their domain name expire.

Matt Kleve: Oh.

Brian Skowron: Yeah, it was pretty rough.

Matt Kleve: So so the website was up. They just couldn't get there from here.

Brian Skowron: Yep. The domain couldn't resolve. Yeah.

Matt Kleve: That's one of those freaky moments, I think, for a representative having to hear the brunt of the angry client who says, oops. Is that how that worked?

Brian Skowron: Yeah. I don't know how they. I don't know how quickly it it took. I wasn't involved in the conversations directly. And I don't know how quickly it took to figure out that it was the domain registration. And I don't know how the how the calls all went down. But the domain registration was not handled by the hosting company. And it turns out that it elapsed because it was in the name of the the team owner's son.

Matt Kleve: Yikes. Yeah, they got their domain back, probably?

Brian Skowron: They did.

Matt Kleve: Okay.

Brian Skowron: Yes they did. I don't know how I, I don't know how domain squatters didn't pick it up, like, instantly. I don't know how any of that went down, but. Yeah. Yeah, they got it back and it was, it was back up and running. You know, by the next morning. But it was a fun story that got passed around a lot.

Matt Kleve: Do you remember, was it in football season?

Brian Skowron: Ooh, that's a good question. I mean, this was over a decade ago, so.

Matt Kleve: I just have to I'd have to imagine that the the traffic to a site would, would be really stronger in season versus...

Brian Skowron: Yeah, yeah.

Matt Kleve: ... in the middle of the summer or something. Yeah.

Brian Skowron: You know, honestly I don't remember.

Matt Kleve: Yeah.

Brian Skowron: I'm not sure.

Matt Kleve: We'll just say it was. The team was making the playoffs. It makes for a better story.

Brian Skowron: Yeah [laughing]. Right, right. It's a bigger fish story. It's right before a playoff game on on on Sunday.

Matt Kleve: Right? so, Brian, you don't know this yet, but I, I scheduled our talk here a little longer because I'm going to ask that you you tell a second story.

Brian Skowron: Oh okay.

Matt Kleve: And it's one that I've heard you tell before. But it's a pretty good story from your web hosting days.

Brian Skowron: Okay, which which story are you referencing?

Matt Kleve: [Laughing] The toaster.

Brian Skowron: Oh, the toaster.[Laughing] The toaster story. Oh my goodness. I'll see if I can if I can do this briefly. Because it's a it's a gem of a story. So, yeah there was a web hosting company that that I won't name to protect the innocent, but they had a you know, small cluster of data, like server racks in, in the office. You know, they use the main data center for, like, a bunch of like, the more critical infrastructure, but there was a cluster of racks that were in the in the office that hosted you know, like, shared hosting clients you know, like the 10-15 dollar a month accounts, things like that. And on a weekend you know, the team, the hosting company would go down to skeleton staff and there would be one NOC engineer, one network operation center engineer because things were typically pretty quiet on the weekends. I happen to come up into the office to work on a proposal on on that weekend. I can't remember if it was a Saturday or a Sunday. But I happened to be up there, and it was me and this NOC engineer, and we'll say his name was I don't know, Justin. So Justin was, you know, working the working he was at the helm you know, doing his thing. And he came, he came by, dropped by my desk and was just chit chatting. And he had he had a loaf of bread, and he was like, yeah, man. Well, I'm just going to go make some toast. You know, because he wanted a snack while he was working, and he so he goes back to the to the break room and I, you know, things were kind of quiet.

Brian Skowron: Then all of a sudden I hear the ambient hum of those those servers in the in the office just sort of go [powering down imitation]. And then Justin comes running out of the break room. You know, he's he's trying to, like, figure out what's going on. The phones start ringing a short while later, he's frantic. You know, things are things are down with those, with those few racks and the shared hosting clients are calling, wondering why their sites are down. You know, our director of of operations comes up in person to the office, sorts it out, they figure out what's going on, they get power back going. They're not really entirely sure why the power to all of these server racks had gone down, and then, I, you know, things quiet down. I'm just working on my proposal. Sort of like watching the watching the urgent chaos unfolding. And then things finally like, you know, calm down. Justin comes back to my desk after a while and he's like, hey man, how's it going? Like, that was pretty crazy. And I'm like, yeah, like he's like, I don't know what happened. That was really, really weird. And then he says, I'm gonna go make some toast. In a short while later, it all comes down again. Because if you haven't picked it up something about the power draw from the toaster was was shorting out the whatever circuit those those few racks were on. [Laughing]

Matt Kleve: No toast at work. There's a new rule. [Laughing]

Brian Skowron: Yeah. No toaster work. No toast at work. It was a good time. This was a I should say, this was a this was a different era. I feel like the hosting industry and cloud services are way more, more battle hardened these days.

Matt Kleve: I'm sure they are.

Brian Skowron: Yeah.

Matt Kleve: Happy Halloween Brian.

Brian Skowron: Yeah, likewise.

[Intro music]

Matt Kleve: So up next we have a story from Nikki Flores, a technical project manager at Lullabot.

Nikki Flores: All right. So yes, Matt, this is not going to be the story about the time I took down the Pantheon site because the dashboard had a modal that was overriding where the database was.

Matt Kleve: Oh?

Nikki Flores: So my team. Yep. This is not that story.

Matt Kleve: Oh that's a different story. Okay.

Nikki Flores: Neither is it the story. When I totally destroyed somebody's Excel file by only selecting the one column to sort by, and it ordered all of the rows and they their look of panic on their face after. And like the heart stopping feeling of despair that I had when I saw the look on their face.

Matt Kleve: I know what you're talking about. I deleted the Lullabot dropbox once. Like the whole Dropbox was gone. It was restored from a backup. But yeah.

Nikki Flores: Oh. Or the time when I got a call. Probably six months, 12 months after I left an organization because I was the owner of the Google Drive. And they said, well, no, I'm not taking down your, your entire workflow. So that was not that story. So this story is from the early parts of my career. And so I'll tell you a little bit about it was we had oh. And just as a, as an extension of this, this is something that I'm much, much, much appreciate here at Lullabot. So, you know, when you're new at an organization or a company, there's a lot of norms that are taking place that oftentimes you don't know what the culture is. Right? So you there's ex- implicit values or just who does work with whom or who feels what about whom. And that's all implicit. It's not necessarily made explicit. And here at this company, I am 100% proud and happy and grateful that we have made the implicit explicit. So we even have documents like it's okay to. We have a lot of things to help people understand what is expected. Okay, so back to this incident. This client was very high profile. So there's a lot of visibility. There's a lot of high expectations on this project. It was a kind of a public trust project. And there's a very, I would say compacted timeline and there's just a lot of scrutiny. I touched a number of organizations. There's a number of people. There are probably over 100 people touched by this project in a contractor fashion or on our team or in the people that we were working as product stakeholders.

Nikki Flores: So the team had a high degree of technical strategical expertise. Everybody was basically groundbreakingly developing the mechanism to do this type of work. I'm trying to be very broad because I don't want it to come back. This was a very big project, and this high profile client had very high profile deliverables, very high touch sort of project and a high level of visibility. So I was tasked with doing what the project, project meetings, regular progress meeting weekly, whatever it was. I had to give the update to one of the active project stakeholders. And I remember Matt, I was giving the update everything I thought was fine, but just out of nowhere it felt like this budding frustration just took root and flowered and burned. And so the regular conversation just quickly devolved into straight up viciousness. So this person cussed me out, right? Like shocked face, sad face, skull face. And to go from regular, like the way that I work is everything is predetermined and you know exactly what to expect. And here's the meeting time and here's what our agenda is. So to have it go off the rails like that was quite horrifying for me. And looking back, this was indeed a high pressure time period. All large projects have recalibration in the lifetime of their project, right? So different pieces coming on, different people coming on, different features. And sometimes there's a bit of discontent with how things are rolling out. But I remember that meeting with my client, cursing at me, and I'm totally clean, like I grew up very straight laced.

Nikki Flores: My family was highly Catholic. I was raised Catholic and almost never heard my parents cuss. I only heard my dad use one cuss word when I was in my mid 20s. Shout out to Florida public school system, which taught me the seven basic cuss words. So thank you. And so to hear these used words, these were words used in a work setting. And then explicitly against me was I was I was so perturbed, like I was disturbed, I felt attacked, I felt worthless, I felt horrified, I felt very small and I was, I went into shock, I stammered, I remember stammering, I said, well, I'll talk to my boss about this. And then I hung up, because this was in the days of phone calls, in zoom meetings and Google meets and slack huddles. Like we do not behave like that because we see people. And I feel like in those days on the phone, you could be a little more removed and that might have been part of it. And I talked to my boss, and my boss was same thing they were shocked as well, and they were a bit resigned because I remember them saying, oh, well, and I'm speaking as my CEO, they're like, well, the client has a problem, so I guess I have to go and deal with it, right? I was like, yep, yes, boss, you got to go deal with it. So all's well that ends well.

Nikki Flores: Shout out to the team and my bosses. They had my back. They worked through whatever the problem was. There was no blame. They were reacting to situation right alongside me. The the project went as as expected, the timeline smoothed out. They made a measurable impact. They got all the accolades. Everything went well. But that instant of just revulsion and horror and shame and fear of what was going to happen, it's just really stayed with me. And I think it brings me back to that quote from Maya Angelou, one of my favorites. I know I've shared it with you before, but people will forget what you said and people will forget what you did, but they'll never forget how you made them feel. And I totally forget what that particular item was, and I totally forget what it was that we even were doing. I totally remember that feeling of just, bam, you are being beaten down. So what I would like to bring away from that is for the rest of my career, I've always tried as much as possible to work towards camaraderie, sense of joint ownership, communication, open communication lines. If people are cussing their cussing at the project or the process or the new tool. They're not cussing at each other. And that's what I love about Lullabot. We have a great deal of mutual respect and collaboration, transparency, honesty, accountability. And in general, I was was in that scary situation and I would never want to be in that scary situation again. And that is my scary story. Tada!

Matt Kleve: I think part of the problem there was that it was such a surprise to you, right?

Nikki Flores: Yeah. When you especially for me, I like to plan things in advance and then having somebody for whatever reason too. And that's the other thing that I have learned from this is regardless of what you're doing in your day, somebody else is going through it for whatever reason. And oftentimes they may bring that forward to your interaction with them. So. I would say it's not scary when you understand that other people are human. I even joke about this with my kids when we see scary monsters on shows or films or something. I'm tell them, oh, they just need some face cream for their wound healing. You know, like, oh, they just need some first aid cream or some sort of Band-Aids and stuff. And so in this case, that person was probably having just a bad day or for whatever reason and took it out on that situation. And so that's the other thing is, as much as you want to put yourself into the work, you do have to remove yourself because it's it's just it's just the work. It's it's just an instance and it will roll on and things will continue. So big picture thinking we got to get it done. We'll get it done on time, under budget, with minimal cussing of our clients and valued teammates.

Matt Kleve: This concludes so wholesomely, like.

[Laughter]

Matt Kleve: This is not like how a scary story is supposed to end. I'm glad I wasn't in your shoes at the time though.

Nikki Flores: Yeah.

Matt Kleve: Thanks, Nikki.

Nikki Flores: Yeah. Thank you. I appreciate it, Matt.

[Intro music]

Matt Kleve: So up next we have senior front end developer Andy Blum. Andy. I'd love to hear your story.

Andy Blum: Yeah. So the previous agency had acquired a number of sites from some other agency in the area. And one of the one of the things that I was tasked with doing at some point in time was, hey, we've got all these sites that when we when we acquired them, we just moved them on to this shared hosting solution. And we basically just copied everything they had and brought it over and dropped it in place. And I was told to move all those to their own individual Pantheon instances. And when I got logged in they were all outdated. They basically had been neglected by the other agency. And so I went in looking at, hey, how do we update these, get them up to date, make sure they're all nice and secure. And at least one of them was older than both Drupal Gettens. And then as I went looking a little bit deeper they all shared database credentials. So they were all had to assume to be corrupt. So got in there and had to go in and separate them all out and do all the updates individually and get them all on Pantheon and up to date and whatnot. But one of those included an Open Atrium distribution that I had to go through and do every update individually to keep the, the distributions, update paths and all that correct, along with all of the other pieces. And that was about three days of work doing that. Tada!

Matt Kleve: You wouldn't want to do that again, would you?

Andy Blum: No it would, it would not be my idea of a good time. And if somebody told me that, I might might have to respond with some some ghost face emojis.

Matt Kleve: I haven't heard of Open Atrium in such a long time that, like, the mention itself, is scary to me.

Andy Blum: [Laughing] Yeah. And it was. It was all Drupal seven. I ended up having to ping, I think Mike Potter at some point was like, hey, is this still a thing? Can this be up to date? And then a couple days later they they came out with a new security update for it. So yeah, it was still a thing. And as a matter of fact, I think there's even an Open Atrium upstream for for Pantheon. So if it's something that you have out there, you could you could put it on Pantheon and it would be easy to update.

Matt Kleve: Very cool. I went to the Open Atrium project page. It's it's still Drupal seven, so. Yeah. Anyway. Thanks, Andy!

Andy Blum: Yep.

[Intro music]

Matt Kleve: Up next we have Brenna Love. She's a technical project manager from LeClaire, Iowa. Brenna.

Brenna Love: So I want to say this was probably 2016, maybe 17. We had an old client of ours who sold religious supplies to, like clergy.

Matt Kleve: Prior to your time at Lullabot, just to be clear. Yeah.

Brenna Love: That's right. Prior to my time at Lullabot. They sold to religious supplies, to clergy, to churches as a place where you could buy altars, you could buy candles, you could buy all kinds of different things. And individuals, it was an e-commerce site, and it was a client that we'd had for boy, I was at my previous company for 16 years, and I think they were a client of ours almost the whole entire time I was there. And around 14 or 2014, 15, 16, somewhere in there, they decided that we or we had had made some changes to how we were managing the business, and we were having clients move on to a support program with us where they were charged a monthly fee. Part of that fee was for hosting and then a certain number of hours for maintenance requests. And they thought that that was too much and they didn't want to work with us anymore. So they they had engaged with a company that was based in India. And of course, you know, a lot of not saying anything bad about all agencies based in India. But sometimes you know, with the time difference with the communication you know, there's communication problems on occasion. Well they, they discovered that this agency was going to charge them far less, much, much, much less to edit their website. And their only complaint was they were sometimes difficult to reach. Well, so they they took the business away from us. We transferred everything over to this new agency, said goodbye. And about a year later, the owner of the company called us in a panic, saying that his site had been hacked. And we were asking him a lot of questions. And as we were on the phone as my old boss and and me, were on the phone and I pulled up the website and it would has been had been completely taken over and it was filled with violent imagery of bombs exploding. There was music playing in the background. There were images of starving children messages of, you know, death to the USA, burning American flags.

Matt Kleve: Not the religious e-commerce site that once was there.

Brenna Love: [Laughing] No. Definitely not. And he was completely confused about why his particular site had been hacked. He'd been trying to reach his new agency for days and couldn't. He had clients calling him, trying, you know, letting him know that they couldn't buy anything. And yeah, it was really, really scary. I've never seen anything like that before. And it took me, like, so by surprise that we were laughing because I was like, this can't be real. And boy, it sure was real. Needless to say, within you know, a few months we had negotiated a contract and, you know, recovered the site, brought it back to our services, and I still think he's with that agency my old agency today, so hasn't been hacked again like that since, but it was very scary.

Matt Kleve: My goodness. E-commerce can be scary anyway. Like, yikes.

Brenna Love: Mm-hmm.

Matt Kleve: Yikes.

Brenna Love: I've never seen anything like it. It was really, really weird.

Matt Kleve: Yeah.

Brenna Love: Yeah.

Matt Kleve: These things are going to keep me up at night. Thanks, Brenna.

Brenna Love: Thank you.

[Intro music]

Matt Kleve: Hello.

James Sansbury: Am I muted or not muted.

Matt Kleve: You are not muted.

James Sansbury: All right!

Matt Kleve: With me now I have director of product Tugboat James Sansbury. Hi James!

James Sansbury: Oh are we actually recording?

Matt Kleve: So James do you have any stories to tell?

James Sansbury: I don't have any stories to tell that that come to mind.

Matt Kleve: So you're really confused why I scheduled this meeting

James Sansbury: Well, I didn't know it was an actual, you know, like, recorded thing, so I thought we were, like, talking about it in preparation for a future podcast in which we would all be on the line together, so.

Matt Kleve: Always be recording. That's what I learned in my news days. So.

James Sansbury: Yeah, I like it. Okay.

Matt Kleve: So, James, I'm going to take you back. It's actually a story of mine that I wanted to tell, and and I wanted to tell the story to you because it's probably way more interesting than me. Just kind of lamenting by myself, so.

James Sansbury: [Laughing] yeah.

Matt Kleve: It's we're going back to June 25th of 2010.

James Sansbury: Wow.

Matt Kleve: What were you doing then?

James Sansbury: June 25th of 2010. I was newly a Lullabot, I believe. And. Gosh, was that like a Drupal Con or something?

Matt Kleve: No, it wasn't a Drupal Con, but I was actually not a Lullabot yet.

James Sansbury: Yeah. Were you, like, a contractor at that point?

Matt Kleve: No, not even yet either. I was a contractor in the fall of 2010, but I was working at a nonprofit organization, and one of my big tasks tasks was working on a membership-based website. It was a Drupal site. It had a lot of media on it, and people signed up and they were able to, you know, get into the website. And the traffic for the website was wild for like two months.

James Sansbury: Oh my gosh.

Matt Kleve: And then after that, not at all.

James Sansbury: Yeah.

Matt Kleve: But it was Drupal six with authenticated users and node access. So what does that tell you? Yeah.

James Sansbury: [Laughing] That tells me you were struggling to keep the servers running, perhaps?

Matt Kleve: Yes. It was very hard on the database.

James Sansbury: Yeah.

Matt Kleve: So. And to try and keep the database going. One thing that I was working on was you know, I was trying to tune the database to try and get, you know, the, the query cache going, like, being as as efficient as possible as well as, you know, lots of other things.

James Sansbury: Right.

Matt Kleve: And there were lots of people in the Drupal community at that time that, you know, had their slide deck of this is how to do it, or I think Lullabot probably did some stuff too. So I think I was tuning a buffer pool or something, changing the size of something. And in order to do that, I had to delete a file on the server and then restart the server, and then it would resize that file.

James Sansbury: Mm-hmm.

Matt Kleve: That's what I remember. I don't remember exactly the details other than I deleted the wrong file.

James Sansbury: Oh no!

Matt Kleve: And I trashed the whole database. It was gone.

James Sansbury: Oh my gosh. No!

Matt Kleve: Moments after that you called me.

James Sansbury: Oh! To see about a job?

Matt Kleve: Yeah exactly.

James Sansbury: I remember this actually. I remember you telling me this.

Matt Kleve: So I just took down production, and, James, you called me for a I mean, it was a job interview that had been scheduled over my lunch hour.

Matt Kleve: And my coworker knew what was going on, and she looked at me and said, Matt, answer the phone. I've got it. It's like, okay, cool. So I proceeded to act like everything was fine for the next hour, and I answered your questions to the best of my ability. But I had to tell you, I didn't use things like panels and features.

James Sansbury: Right.

Matt Kleve: Because because I was at a small organization and bringing layers of complexity like that wasn't what I was trying to do. I was trying to make things easy.

James Sansbury: And even saying panels and features, like probably many listeners are like, I don't even know what that is. But back then, that was like the budding, you know, best practices for Drupal. So.

Matt Kleve: So you were like, do you use panels? I was like, no. But but here's why. How about features? No, for the same reason.

James Sansbury: Yeah.

Matt Kleve: And then I was like, well, crap, I didn't get this job because I don't do any of this stuff that they wanted me to do. But turned out to be all, okay. I had a backup, and I got a job, and that was probably my my most terrifying few hours of web development.

James Sansbury: Yeah. And what's funny and fun about that story is that, you know, if you if that scenario had fast forwarded, like, maybe a year or two. One of the questions that I liked to ask during interviews later was, tell me about a big mistake you've made and how you handled it.

Matt Kleve: [Laughing]

James Sansbury: And I'm trying to imagine what might have happened if I had asked you that question in that moment.

Matt Kleve: Poker face.

James Sansbury: You'd be like, well, funny you should ask. I'm not quite sure how I'm going to handle it, but here's what's going on in the background.

Matt Kleve: [Laughing] Yep. So yeah, my data was I mean, I had a backup. I probably lost a day's worth of data, but I got it back and up and going.

James Sansbury: Oh that's good.

Matt Kleve: Shortly after my job interview and yeah, I eventually got a job offer and was a contractor for a while, and I've been here for like 12 years now, so it's good.

James Sansbury: Yeah. That's right. It was. I'm glad you took the call. And and you know, the the really scary ghost story would have been if you didn't have backups, right?

Matt Kleve: That's true.

James Sansbury: Because I have done that before, and that's how we learn to how important backups are is by those experiences. But yeah, that's a fun story. Oh man. Yeah. I'm actually like, now trying to think of something scary.

Matt Kleve: You could tell the domain story. Shoot. That was a bad...

James Sansbury: Oh gosh , that's right!

Matt Kleve: ... week or two for you.

James Sansbury: Yeah, indeed. That might take, like, a whole podcast to really describe that, though. Yeah. You know, I mean, the the short story of it is that in early on in Tugboat days, we thought it would be cute to have a .qa domain because the service was Tugboat QA. And that is a country code top level domain.

Matt Kleve: Qatar.

James Sansbury: For for Qatar. Yeah. And and, you know, so we registered the domain and that was, we put everything there and that worked for many, many, many, many years. Probably like 5 or 6. We had that domain as our primary domain. And then one day I woke up and it just didn't work anymore. [Laughing] It's like what happened? We've we just renewed the domain. Like, I know we had auto renew on, and I even double checked it, and and it was just gone, and we called our registrar, and they wouldn't let us talk with anyone. They were like, yeah, just keep checking back. We're working on it. And, you know, repeated calls, repeated calls, even got our lawyer on the phone to call them. They're like, you know, sorry about that. We're working on it. The engineers say they're working on it. And turns out that our registrar, months prior had been refused by the actual country for continuing to provision domains using that country code top level domain. And they hadn't told us that. But they couldn't they like, literally could not renew it because they were no longer allowed to. And so after working directly with the, the, the, I forget what the name of the agency is from the country of Qatar who the extremely friendly wonderful folks. They explained the situation and they were like, yeah, you'll just have to register it with someone else and we'll transfer the domain to another registrar for you.

James Sansbury: And so, gosh, it was just like it was almost a full week of not having our domain. And then finally, you know, calling the country, calling the, the government, you know, and saying, like, please help us because our, our registrar is not helping us. And then they finally, you know, like we got it sorted out and everything. And in the meantime, you know, as you might expect, we've now gotten away from using the vanity sort of URL, top level domain, country code thing, and we probably won't ever go back. So definitely a lesson learned there. That country code, top level domains are complicated, and there's a lot of, like things that go along with it. So be careful if you're hosting using that. And, yeah. And and it was definitely like the stuff of nightmares sort of a thing.

Matt Kleve: So, so Tugboat.qa Now resolves to TugboatQA.com.

James Sansbury: Yes. Yep. And as you might imagine, like I have, I have a as I was going through all the tasks to transition to a new domain, I was like writing everything down that I had to do. And I have, like, a checklist that I went through that, you know, has like hundreds of items on it when it came, when it when we finally got it all done, it was like hundreds of things that needed to be done. It was just such a monumental task to switch to a new domain. I mean, just think about like email and all the different services that emails are connected to. And, and, you know, that was one of the other real nightmarish thing about it is that all of our email was down. Like we had @Tugboat.qa emails. And so our customers were trying to reach us and they couldn't reach us because email wasn't working. So we were like, getting direct messages on Twitter and stuff. And you know, those fun times. We can laugh about it now, right?

Matt Kleve: Yeah, something like that.

[Laughter]

James Sansbury: So, yeah, there's my there's my spooky story for you, Matt.

Matt Kleve: Thanks, James. Happy Halloween.

James Sansbury: Yeah. Happy Halloween.

[Intro music]