Inside Lullabot's Support & Maintenance Department

0:00 0:00

Mike and Matt discuss various aspects of Lullabot's Support & Maintenance Department, and how it differs from Client Services, with Director, David Burns, and Project Manager, Cathy Theys.

Episode Guests

David Burns

David's career in Drupal spans over 18 years. His passion for business and technology led him to build Lullabot's Support & Maintenance department where he serves as the VP of Extended Support.

More about David

Cathy Theys

Cathy is the Support Team Manager at Lullabot, a natural problem-solver, and a Drupal contributor.

More about Cathy

Transcript

Matt Kleve: For April 1, 2021, it's the Lullabot podcast. Hey everybody, it's the Lullabot podcast, episode 252. I'm Matt Kleve, a senior developer at Lullabot. With me as always, cohost of the show, senior front end dev, Mike Herchel. Hey Mike.
Mike Herchel: Hey Matt, how are you doing?
Matt Kleve: I'm pretty good. We're going to do this thing again, it's been a while.
Mike Herchel: It's been too long.
Matt Kleve: Let's get the band back together.
Mike Herchel: Good idea.
Matt Kleve: Google tells me there are millions of Drupal websites on the Internet, and we at Lullabot know about a few of them.
Mike Herchel: We do.
Matt Kleve: Yep. And Google also tells me that Drupal 9 was released on June 3, 2020.
Mike Herchel: Sounds about right.
Matt Kleve: I'm going to guess that all of those millions of websites are running Drupal 9 and the latest greatest hotness of Drupal leading edge code.
Mike Herchel: I would maybe step back from that one, Matt.
Matt Kleve: We're in a state where there are Drupal 7 sites that exist, there are Drupal 8 sites that exist, there are Drupal 9 sites that exist, and they're all different in their own special way, and they're all actually technically supported and up to date at this time, right?
Mike Herchel: Yeah. So how do we handle this with so many different versions?
Matt Kleve: We have some friends in Lullabot that do that for us now. We've got people for that, right?
Mike Herchel: Yes.
Matt Kleve: We do. On the podcast today, we have our Lullabot's director of support and maintenance, coming to us from Philadelphia, David Burns. Hey Dave.
David Burns: Hey Matt. Hey Mike.
Mike Herchel: Hey, how are you doing?
David Burns: I'm doing well. Very glad to be on the podcast with y'all today.
Mike Herchel: Thanks for being on. Also with us is our support and maintenance project manager, Cathy Theys, coming from somewhere Virginia, Washington D.C.-ish, right?
Cathy Theys: Yeah, I like to keep it vague.
Mike Herchel: Yeah, you're a little bit of everywhere. How are you doing today?
Cathy Theys: I am doing super. I'm really happy to be talking about maintaining Drupal sites.
Mike Herchel: It's an important topic.
Matt Kleve: It is an important topic. A lot of times everybody talks about people implementing the latest, greatest, the new APIs, the best way of building new things with all of the coolest new tools, but at some point all of that new stuff gets put into use and not everybody is undergoing a huge website build all the time, and that's why the Lullabot support and maintenance team exists, right? How long have you been doing that?
David Burns: So officially, Lullabot support and maintenance, I looked at the existing SOWs, began January 1, 2019, so really only two years ago.
Matt Kleve: And what kinds of things do you do?
David Burns: Well, before we get into that, I'll just give a little more background than just the start date. So the reason that we got into support and maintenance is because we had a few people that were working on projects and Lullabot tries to have one developer on a project for full-time, like that's the only thing you work on and it can be for three months, six months, however long it takes to build that project.
Matt Kleve: For example, Mike is a front end developer and Mike you're on one project, right? You're full-time committed to the build out of this one project.
Mike Herchel: Absolutely.
Matt Kleve: I'm on a project, I'm full-time committed to this build out of this one project. But that's harder when you were trying to do support, right? That's what you're trying to say?
David Burns: Yeah, it's much harder and that's kind of why the department came into existence. So we started getting inquiries that like hey, let's have you work on this site that you've already built, stick around, hang out with us and fix bugs and implement new features. And we started taking on some of that work, but it wasn't officially under a specific department. And I was brought in to assist the people that were doing that type of work at the time and it was working well. So Seth and Matt and people from the leadership team reached out, it was like hey, there's a need for this in Drupal and there's other Drupal shops doing this, so let's commit to doing it. And that officially gave the Lullabot support and maintenance brand.
Matt Kleve: So can you talk a little bit about the mindset that might be a little bit different to somebody's who's doing maintenance on a Drupal site?
Cathy Theys: I think one of the things that we think about is the scope of the request from the customers. And so we're like is this a maintenance type task, is this a security update, is this a regular kind of update, is this a bug fix, is this a feature request, how big is the feature request, what exactly are they asking for? And so I think the overall mindset is always wanting to do the little maintenance tasks to keep things running smoothly and nicely, but keeping your eye out for requests that are more than they seem to be.
Mike Herchel: And that never happens with web development, does it?
Cathy Theys: Never.
Matt Kleve: A lot of times people talk about the metaphor of building a Drupal site with Lego, right? You have a bunch of Lego blocks and you're building this huge house or something. And I think a lot of times when you're maintaining a site or working with an active site, long term, swapping out that one blue block might be a little bit difficult too, because of all the other things that it's resting on it would seem, right?
Cathy Theys: Yeah, you need to be careful. We have a lot of tools that we use to catch unintended changes. And helping our customers utilize those tools and bring those into the project is something that we do that might be like a big project, but its goal is to increase automation to make maintenance easier. So we do some automation to help catch unintended consequences.
Mike Herchel: Let's jump into ... This is one of the questions that I had for you right here is you have updates, you have dependencies, you have imports. What types of things can you automate, when does it go wrong? What type of things are still manual?
David Burns: So one of the things that is challenging about support is the context switching, the amount of context switching we do, right?
Matt Kleve: Which is kind of why developers who are on full-time projects, like Mike and I, don't end up changing projects. That's something that Lullabot has carved out in saying, "Context switching is hard, it's best if our people can stick to one thing," but that's not what you get to do, is it?
David Burns: Not exactly.
Cathy Theys: No, we get to specialize in context switching. That's like-
Matt Kleve: Well, that's fair. That's fair and that's an important skill.
David Burns: I like that. So one of the things that we do to help reduce or streamline context switching is we standardize on stuff, right? So there's DDev that allows you to have a config file and you run a command and it spins up in local site.
Matt Kleve: It sits in front of Docker, right, DDev?
David Burns: Yeah, DDev's built on top of Docker. So that's one system, but we started using Lando way early on, like one of the first-
Matt Kleve: Which is the same idea, right? I've used both in consecutive projects and I couldn't tell you the difference other than I used a different command to do stuff.
David Burns: Under the covers, under the code that's running, it's all Docker based. It's all the different layers that get added on top of it and the command you can use and the pre hooks that exist that let you do more advanced things. So as a department, we decided we're going to use Lando, and what we've done, we started out with Lullabot.com, which is our own site where we get to explore and test and figure out scripts-
Matt Kleve: Which Lullabot hires you as a maintainer for Lullabot.com, don't we?
Cathy Theys: Well-
David Burns: I'm not sure if hire's the right word. It's like you do maintenance and we're sending you a paycheck, so take care of this thing that our whole company cares about.
Matt Kleve: Well, we do, yeah.
David Burns: And I'm totally on board with that.
Matt Kleve: Okay. Okay, that's good. Sorry, Dave.
Cathy Theys: Yeah, but it's a great guinea pig though and we're really empowered to try out things and it's a very safe environment for us, because we know that if we impact the customer, we're impacting us and we'll fix it right away. So it works out really well to let us test run things that we might want to maybe future standardize in support, we can try it out on Lullabot.com first.
Matt Kleve: So you were talking about putting everything into Lando. So if you were starting with a new client, do you end up then putting all of their stuff into a container, into Lando and going from there?
David Burns: Yeah-
Matt Kleve: Is that step one for you because you need to specialize your process?
David Burns: So some of the projects come to us already having these type of tools implemented. Some of them may be using VirtualBox, the VM system, and we'll take a look at that and be like, is there enough reason to stick with doing that and what is the time it takes to just swap in all the fancy good stuff that we know works well with Lando? And I would say 9 times out 10, we wind up integrating Lando.
David Burns: We do some really neat stuff that when you do the start, it'll go ahead and fetch the latest database and not import it, but give you a command in the output and says, "Just paste this and you'll get a fresh copy of the site running locally." We do fun stuff for some clients where we actually compile themes when we run Lando start. We were noticing that some of the clients were coming to us and be like, "Hey, this page looks different than the live site, why is that?" And we would get up and say, "Hey, remember to build the theme."
David Burns: When you start repeating things over and over again, you're just like let's script this, let's have somebody fresh to the project be able to come in, run one command, and basically have the whole site running, and that's what Lando's been able to provide for us.
Cathy Theys: Yeah, it's a really good investment that we will look at, because sometimes like Dave said, we'll keep what automation the customer already has. But if it's compelling or if they don't have one and we set up Lando, that investment really pays off later, because we have people switching projects. Even if they stay on that project, they're still going to be switching in and out of it, because they're switching between three projects.
David Burns: Yeah, it's not uncommon for me to have a day where I'm running Lando Start on four different sites.
Matt Kleve: I'm going to guess you need a decent sized hard drive for all that.
David Burns: Yes. We have some massive projects, Georgia.gov, GovHub is one of them.
Matt Kleve: Which was a Lullabot project that we then passed off into the support world, right?
David Burns: Yeah, and that's how we get a fair amount of the work within support, I think over 70% is passed over from client services. But we were noticing that when we were onboarding new people to the GovHub platform, we'd have to bump the default Docker setting from 80 gigabytes of space up to 120 gig, just because how massive the platform is, how many databases it's pulling in.
Matt Kleve: And that's not a Docker issue, that's just the website is huge, right? It's a big multi-site website and yeah, there's just a lot going on there.
David Burns: Exactly.
Mike Herchel: So outside of local environments, is there any other automation and, or standardization that you can do across the different projects?
David Burns: That's actually been a big push as of the last quarter of last year and a major focus of this year. We have a lot of clients that are on Pantheon, but we also work with some that are on Acquia, some that are self-hosted. And the thing that allows us to context switch quicker, reduce the amount of errors is to after you're committing code locally and pushing up to be able to build environments and see the work that a developer's doing, and a Lullabot built tool called Tugboat QA is perfect for that. That integrates with GitHub, GitLab, Bitbucket, even privately hosted repos.
Matt Kleve: Tugboat.qa is the website, and we'd done a previous podcast with the Tugboat folks, Mike.
Mike Herchel: Yeah.
David Burns: And what's great about Tugboat, it has integrated the Lighthouse integration for Google that gives you SEO scores, performance scores, accessibility scores, at a very general level, you still want experts to go in and do the fine detail stuff. But we're able to look at that, we're able to look at the visual diffs that come with Tugboat. We provide one of each content type, the major landing pages. We even have a workaround that allows it to screenshot and do visual diffs on node add, node edit, and editing pages, stuff like that.
Mike Herchel: That's pretty cool. Can you just explain for our listeners what a visual diff is really quick?
David Burns: Sure. So we have this concept of base previews, basically what does your live site look like right now. And then you make some code changes, you commit it to Git, Tugboat spins up an environment. And then Tugboat goes hey, let me take a screenshot of this new PR that you've been working on and I'm going to overlay it with a screenshot of that base preview and it's going to do a pixel by pixel check looking for any variations.
David Burns: And Tugboat's able to say this page is like 98% perfect, and sometimes these slight variations are a different image showed up or something shifted. Anything above 90, I wouldn't be too, too concerned about. But once we start seeing numbers in the 80s and 70s, we're like hold on, we definitely don't want to merge this in, in its current state, let's take a closer look at that specific page and figure out what's going on.
Cathy Theys: Yeah, it's really useful.
Matt Kleve: As you know Mike, and I'm sure Dave and Cathy, CSS cascades and a small change here might affect elsewhere, right?
Mike Herchel: CSS-
Matt Kleve: And so a visual diff would super be handy to do that, and that's what I was thinking about. I know that a lot of times when I come on to a new project, even if it's not a support one or they have an existing website, I get a ticket to change something and then I start working on it and I'm like I hope I'm not breaking the rest of the website with this change, but I think I'm doing the right thing. There's always that hesitation I guess, and so having these tools set up I'm sure help out a bunch, and not just you on the support and maintenance team, but the client as well who can leverage that too, right?
David Burns: Yeah. We could send a Tugboat link for that PR environment, they could click around as if it was the live environment, they could add content, remove content, anything they want to do and give the thumbs up, hey, this is ready to go. And then what's really neat is once they click the hey, this is ready go, usually a lot of tickets would stack up in the course of a sprint or a couple weeks and they'd tag a release and deploy it.
David Burns: We're starting to look at the GitHub Actions tool that basically says once you merge this into whatever you're main branch is, we're going to see GitHub or whatever send off a web hook and be like this is okay and approved and we're just going to push that out to the dev environment. So we always have an environment that's the latest and greatest code.
Cathy Theys: Yeah, and usually that dev environment is usually then on the customers hosting infrastructure, not on Tugboat. So that goes through Tugboat and then moves into something that's closer to what will actually be deployed with the live site.
Matt Kleve: Does hosting matter? Are you hosting agnostic, or do you want to support these two things?
David Burns: We have preferences, which reduce context switching and it's good when we have a lot of clients using the same hosting, but right now we are okay and we accept any enterprise level hosting or anybody that's doing custom hosting.
Matt Kleve: Sure.
Mike Herchel: So if I'm using, FTP you won't support me?
David Burns: Nope. We need version control, that's where we're at these days.
Matt Kleve: FTP on a GoDaddy, right?
David Burns: Mm-hmm (affirmative).
Matt Kleve: GoDaddy-
David Burns: And I just want to be real clear, our primary purpose within support and maintenance is not to be system architects. We take care of the Drupal side, the application side. We will happily reach out to the support team of those hosting providers if there's performance issues that are not caused by the application and coordinate a fix with them if the support client doesn't feel comfortable doing those things.
Cathy Theys: Yeah. So we work with all these different hosting companies, a lot of the common Drupal ones and custom things that customers have, but that gives a wide experience set. One of our customers can say, "Hey, I'm seeing this weird issue," we go through the catalog of all the other customers that we've had and the ones on all these things and we can quite often be familiar with that host and maybe that specific issue, because we've worked with the host on a different customer earlier. And so we can really help out smooth some of that over and make those resolutions come quicker for our customers.
Mike Herchel: Makes sense. So Dave, you mentioned automating things with GitHub Actions and tools like that. On the Lullabot.com repo, I see all these notifications come in for Dependabot, to the point where I actually set up a little filter, so I don't have to look at those. Can you talk about maybe your dependency automation and maybe other things that you might be doing?
David Burns: Yeah. So a lot of our support clients have limited hours that are available to us within a month, and early on we noticed that somebody that's with 30 hours a month, a majority of our time would be spent writing a Composer update for a specific module, going in, testing that change and things like that. And then I heard about Violinist IO through Twitter and I was like this is the greatest thing ever, one pull request per module update and you get to layer on-
Matt Kleve: I don't know anything about it, Dave, can you explain that to me, Violinist?
David Burns: I'm sorry, what's that?
Matt Kleve: Violinist.io, I don't know anything about that.
David Burns: Yeah. So all it really does is look at your composer JSON file and it runs a Composer update dry run, looks for anything that is out of date or there's an update available. It also has some options around do you only want to create a pull request for modules that have security updates as opposed to minor point releases. And when it sees one of those, it'll go ahead and create a pull request for that. And included in that pull request is some of the change logs if the module maintainer has been good about that, and it'll print it right in a comment or the description of the pull request and be like, here's all the commits.
David Burns: And with all things that we have layered on top in Tugboat, we automatically get the visual diffs, the Lighthouse scores and everything, and then we're also experimenting with automated tests. So if everything passes after Dependabot or Violinist IO creates that pull request, we're feeling pretty confident this is safe to merge in. We'll go in and look at the config sync page to make sure there's no overwritten config. We'll click around on the places we know that that module interacts with the site to just give us a lot more confidence.
David Burns: But in setting up tools like Violinist and Dependabot, we found that what used to take maybe 30 hours a month for our team to work on, we're really only spending five. That frees up 25 hours to get to know the client, their project, the things that they're struggling with their editorial system, or let's start talking about bugs or small feature enhancements. So we're providing a lot more service by layering in these automated tools.
Matt Kleve: Those sound like really great tools for you, but they sound like really good tools for anybody who runs a Drupal site.
David Burns: Definitely recommend going and checking them out, see if they're a fit for your organization or your own personal sites.
Mike Herchel: We're adding the link for Violinist.io to the show notes.
Matt Kleve: We're talking to David Burns and Cathy Theys of the Lullabot support and maintenance team. After this, we'll talk a little bit more about things that they find when they dig into Drupal, coming up right after this.
Mike Herchel: Welcome back. We're talking to the crew from Lullabot support and maintenance department. So as all Drupal developers know, there's a million ways to do anything in Drupal. Is there anything that really, really stands out where you looked at something and you're just like oh no, or you're like, I should charge these guys double, or something like that?
Cathy Theys: What have they done?
Mike Herchel: I know.
Cathy Theys: Why?
Mike Herchel: Any SQL queries in the template files or anything like that, that you've seen, that you can think of? You don't have to name names.
Matt Kleve: Can you still do that in Drupal 8?
Mike Herchel: Not in Drupal 8, but you can in Drupal 7.
Matt Kleve: I'm doing all kinds of awesome Java script right now on a Twig file.
Mike Herchel: You shouldn't admit to that.
Matt Kleve: I only say that to make the front-ender's blood pressure go up. But Dave and Cathy, I can see by your 1000 yard stares that everybody's website is a little bit different.
Cathy Theys: Yeah, I'm trying to think. Usually, things are a little different customer to customer, but it's more interesting than shocking, and so it might be ... I think we approach it as an opportunity to figure out how does it work and why did they pick it this way, because we need to know that in order for us to have confidence that when we change it, we're changing it in the way we intend.
Cathy Theys: And so, quite often we'll see things that we don't understand and we'll just be like what is this? But then we talk to our coworkers and we talk to people in support and we talk to people in client services, and sometimes I talk to Drupal people in Drupal.org Slack about things and I'm just like, hey, I'm working on this thing and I don't know anything about it, can you help me please figure it out?
Matt Kleve: The hive mind is pretty great working within Lullabot, and of course being a part of the Drupal community, there's always that place you could reach out and that's cool.
Mike Herchel: Yeah, totally.
David Burns: I would think that one of the things I say is we live in a world of riches when it comes to support and maintenance. We are given these projects that client services team has worked on, built from the ground up, and we see things like in a Lando file or the way they're compiling things and we're just like, this is totally useful, we're going to mark this down and try to reuse it as much as possible, again, working towards standardization to reduce context switching.
David Burns: So that's been really, really helpful and we'll even wind up in situations in support where they need somebody that had built the original site to do a specific larger task that's outside of the scope of support and maintenance. So working directly with Lullabot, we're able to say, "Hey, come in on this same support SOW and address this feature." It doesn't happen as often as I'd like, but when it does happen, it's so great to be able to see the cross sharing of people, between both teams.
Mike Herchel: I remember I came in and I did a small, what was it? Like a one week project where the goal was just to make this navigation menu for an EDU accessible. It was one of those situations where the drop downs were not keyboard accessible, and so I'm like, all right, I could do this. And it was fun because I was doing stuff like that and I got it working within two days and I was like well, I could do these other things, too and they were like yeah, and that was a lot of fun.
Matt Kleve: These-
David Burns: Yeah, that's ... Sorry, Matt.
Matt Kleve: These quick contracts aren't necessarily normal, but it's fun when it works out, Mike, right? You've got some availability-
Mike Herchel: Yeah. I was in between client projects.
David Burns: Yeah, we've been reaching out frequently for accessibility audits, which has been really insightful, definitely something I'm wanting to learn more about, and menus keep coming up all the time. Menus and accessibility are like the number one thing that we're having to reach out to get some super-duper experts outside of support maintenance to help out with.
Matt Kleve: Side bar, Mike. We've got a bunch of new accessibility experts. It seems to be there's been a really strong resurgence inside of Lullabot and I know we've done a ton of podcasts about it, but we should think about doing that again?
Mike Herchel: Yeah, totally. I actually know who I want to invite. I'm not going to announce this person right now.
Matt Kleve: It's a secret.
Mike Herchel: Yeah, but I am friends with them. Well, they wouldn't call me a friend, but I would call them a friend.
David Burns: Mike, you've got so many friends.
Mike Herchel: They would be like, stay away.
Cathy Theys: I'm trying to think what unusual things do we see in support. And the other thing that I thought of was I see a lot of interesting things because some of the sites that we work on have been around for a very long time. So it could be like 8 years, 10 years. I don't know how long has Drupal 7 been around, right? It's just been ages, and so I can see the timeline of the way the popular things have changed in Drupal, because on one site, it might be like compiling SASS and be using Compass. And on another site it's like pure CSS and I can see different stages of things.
Cathy Theys: And so we do see a lot of wild differences, but they made sense for that project at the time under the constraints that they were on. They were just doing the best they could whenever they had the development budget and they were working on the site and they were like, well, we need to do this quick, or we need to do this rigorous. Whatever their constraint was, they did the best they could. And then their site's been round for ages and ages and it needs maintenance and they come to us, and we're just like, we understand.
Matt Kleve: I like your attitude. I've come around to that in the last few years too. It's the not what the hell was this person thinking, but what caused this person to think that way, because there was something going on at that time and this was the right way. You give them a little more grace, right? It's like the person before me, they were in a situation and that might not make sense now, but yeah.
David Burns: It all boils down to time and cost, right? We'll see projects and we'll be talking with the customer and they're like, this thing we built two, three, four years ago is just so troublesome, and just talk-
Matt Kleve: And it was a temporary solution two or three years ago.
David Burns: Yeah. So one of the things we'll come do initially is an audit and make sure the site's using best practices, making sure modules are in the right places, make sure there's a patch Makefile for D7, make sure you're using Composer for D8, and just get us to a baseline that we know when we bring anybody else into the project, they're seeing less and less of the surprises.
David Burns: So that is part of the standardization and cleanup thing we're focusing on, and sometimes the challenge there is we want to do these things that provide everybody a better quality of life, but the client is wanting these many features to be focused on. So it's a constant balancing act of we're going to deliver the thing that obviously you're paying for our services, but we also need to take care of these things to make everybody's life easier, not just the Lullabot team.
Matt Kleve: You were talking about how long it's been since Drupal 7 came out, but Dave, I know you and I both have built Drupal 6 websites for Lullabot, so yeah.
David Burns: There are no Drupal 6 sites right now that I can think of in support and maintenance and thank goodness for that, because-
Mike Herchel: Would you take a Drupal 6 site?
Cathy Theys: Oh my gosh, I don't know.
Matt Kleve: 6 to 7 is a whole lot quicker migration than 7 to 8, but I know that a lot of people are wanting to go 6 to 8, or 9 I guess at this point.
David Burns: I think it would be hard for us to flat out say no, we wouldn't accept some at some point. We're always going to try to do an audit before we say yes, and if we're feeling good and we're on the same page with the direction the site owners want to go, like hey, it's on Drupal 6 now, but within a year we plan to migrate Drupal 8, Drupal 9, those things are taken into consideration.
Mike Herchel: Yeah, that's makes sense.
Matt Kleve: After that conversation, if it's something that you don't feel comfortable undertaking, do you have friends or enemies you refer them to?
Cathy Theys: There are companies that were approved or advertised Drupal 6 long term support vendors, they're called something slightly different. Yeah, we might refer to a friendly competitor, because there's some people, they did that, they're good at that.
David Burns: And they still do that.
Cathy Theys: They should keep doing that, they're great at it, but it's different.
Matt Kleve: I would think that newer versions of PHP would start to become the issue with running those older versions of Drupal, because you find a new host right now, they're going to run PHP 7 point something and Drupal 6 might laugh at that.
David Burns: And that's one of the things we would look for, like where is this site hosted? Chances are Drupal 6 might be using SVN instead of Git. So I don't think we would want to add the overhead of having to have every person on our team remember all the SVN commands.
Matt Kleve: That wasn't that long ago, Dave. You and I have been on a Lullabot project together that used SVN.
David Burns: Absolutely.
Matt Kleve: It's something that is in recent history if we want to call it that, but shoot, that's been like nine years.
Mike Herchel: I wouldn't call that recent, Matt.
David Burns: Not [crosstalk 00:33:05].
Matt Kleve: I don't know, is Drupal in your recent history, Mike? It depends on I guess your perspective. But Cathy, go ahead.
Cathy Theys: Oh no, I was just going to go back to the Drupal 6 question.
Matt Kleve: Oh go ahead, yeah.
Cathy Theys: I think we would raise our eyebrows at that, but we do have Drupal 7 customers and we are an official, approved Drupal 7 extended support vendor.
Matt Kleve: More to come on that later, right?
Cathy Theys: Yes.
Matt Kleve: Well, the end of life for Drupal 7 is in November '22, so we've got a little bit left that you can still use Drupal 7 and feel mostly okay with it. Drupal 8's end of life is 2021 though, right, this year in November?
David Burns: Yeah. So the reason for that is the version of Symfony that is shipped with Drupal 8 is going to go end of life, and Symfony is not in Drupal 7, so we have to follow where Symfony is going. I do think it's weird that a newer of version of Drupal is not being supported as long as an older one, but I understand how they got to that reasoning.
Matt Kleve: Sure. Well, so they'll upgrade.
Cathy Theys: Yeah. It's a lot easier to migrate from Drupal 8 to Drupal 9.
Matt Kleve: Kind of like I was talking about 6 to 7, I think 8 to 9 is probably about the same as 6 to 7 was.
Mike Herchel: I would say easier than that. I mean, a lot-
Matt Kleve: It depends on what your custom code is, right? If you were doing bad things in your custom code in Drupal 8, then it's going to be harder, right?
David Burns: I think 6 to 7 there was a bit more schema changes because we were starting to explore the concept-
Matt Kleve: And I remember, didn't renderables became a bigger deal at that point, or am I wrong? I thought there was something in the theme layer that became trickier between 6 and 7, but it's been a while.
David Burns: It's so long ago.
Mike Herchel: Yeah. We should do a podcast on that, Matt.
Matt Kleve: Okay.
Mike Herchel: The 6 to 7 upgrade.
Cathy Theys: What?
Matt Kleve: I'm sure there's actually probably one out there, right?
Mike Herchel: I know, right?
Matt Kleve: Like the old Jeff Robbins, Matt Westgate podcasts of 10 years ago.
Cathy Theys: I know we're laughing, but I'm trying to think in my head, would that ever being a thing somebody would want to do, migrate from Drupal 6 to Drupal 7. And the only reason I can think that somebody would want to do that is maybe if they wanted to migrate to Backdrop, because Backdrop is kind of like Drupal 7, and-
Matt Kleve: A fork, right? It's a fork, it's a friendly fork. It's not a fork like the open source community's angry with them. They're providing-
Cathy Theys: Right. It was forked from Drupal while Drupal 8 was under very intense but prolonged development, and so it's kind of a snapshot of Drupal 7. It's got the best things of Drupal 7 in it, but it's got the features of Drupal 8, but the code underlying it is different. And so if it was too expensive to migrate from Drupal 6 to Drupal again, like to Drupal 9 would be where you would migrate now, and that's too expensive, somebody might want to migrate to quote "Drupal 7" except that Drupal 7 that they migrate to is really Backdrop.
Matt Kleve: That's interesting. I Googled something real quick, and so there was actually a Lullabot podcast, Drupal 7, Are You Ready? Episode 90.
Mike Herchel: Nice.
Matt Kleve: Mentioning that in an article even more interesting. There was an upgrading from Drupal 5 to Drupal 6 that came out over 10 years, that's a really great article that's written by Mr. David Burns.
David Burns: I have Lullabot.com articles? I need to start writing some more of those. But while we're talking about the upgrades, recently we have started focusing on the Drupal 8 to Drupal 9 upgrades within support. That's one thing that we do because the upgrade process is so much easier than it has been in the past. One of the things we do is drop in the Drupal Rector module and the upgrade status module, and we run an audit report to give us an overview of the level of effort. It's basically upgrade your modules to versions that support D9, that's fairly obvious.
David Burns: Drupal Rector will scan your custom code and be like this is deprecated, go ahead and replace this with this. And it even ships with a command you could run that's like, we'll try to fix as much of that as we can automatically, and I found it works for about 80%, if not more, of the code that needs to be changed. Getting the D9's a much, much more enjoyable experience than going from Drupal 7 to Drupal 8.
Cathy Theys: Yeah, and for some sites it might be a maintenance task, and for other sites it might be oh wait, we need a dedicated person for this. This is a little bit more customized and needs some more attention, in which case we would talk to the customer about that and maybe hand the project through our sales team back to client services.
David Burns: Even right now with a project like GovHub, which is by far the biggest Drupal site I've worked on in terms of amount of code and sites that it's running, if the SOW has enough months and the client's willing to work with the support team in a long enough timeframe that we could do sprint models, where we do five modules here, until we get everything done, that can still land in support, it's just how quickly does the client need to be on Drupal 9. And if it exceeds our quote-unquote "support bucket" we would send that back over to client services.
Cathy Theys: Yeah, that's a good point.
Matt Kleve: What does your typical contract look like?
David Burns: So it breaks down into three tiers. The entry level, which is like 30 hours a month, and that really focuses on the initial security audits and recommendations, performance audit, module and security updates, integrating Dependabot, and then just critical bug fix, it's like this is affecting the user experience. And we find that 30 hours a month, 30 to 50 is a good place for those type of projects.
David Burns: And then the next tier would basically be like a consultation and mentorship. It includes everything that was mentioned in the first tier, and then on top of that, we do implementation and architectural guidance. So they'll reach out to us and be like we're looking at these three modules, which one would you recommend if we're trying to do X, Y, Z and we'll give the pros and cons of each. We'll also be able to do pull request reviews, a little bit of project management and road mapping, and that's for clients that are like 50 to maybe 80, 90 hours a month.
David Burns: And our third tier is basically dev augmentation. A lot of the clients we work for have small departments, like one or two people and they are hiring us just to do continued development to help augment their existing team. It includes everything from the first two tiers that I mentioned, but also includes maybe one developer part time, or two to three developers part time depending on how many hours are available. We'll also do ... We'll set up more robust automated tools, we'll probably do some Cypress IO testing or Nightwatch. Is that what it's called, Nightwatch JS that's in core?
Matt Kleve: Sounds good to me.
Mike Herchel: Yeah.
David Burns: So those are the three tiers and our rates are similar to the hourly rates that any Lullabot project would have, just because the type of work we're doing is support, we still operate at the same cost for Lullabot.
Cathy Theys: Yeah, I like it when our contracts are long and retainer-based and are big enough to have multiple people on it. Those are my favorite ones.
David Burns: Those are the ones that allow us to really get to know the platform, really get to know the client and their team, and then really talk to them about what their experience is with the site that we've built, or the site that we're maintaining for them. And those feel much more like partnerships where we're bouncing ideas off each other trying to reach a common goal, where the other ones, while they're great, are just like we've got the feature set or this site's not going to change much, we just want to make sure there's no security flaws that people are going to take our site down.
Cathy Theys: Yah, they're different. But they're also like it's whatever the customer needs at that point, right? When they need and want that collaboration and investment and exchange of learning and improvement of their infrastructure, we're going to be excited about that. But when they don't need it, and what they really need is just bare minimum, maybe because of the project lifespan or maybe because of the budget fluctuations, we're going just help them do whatever they need to do with whatever they got.
Matt Kleve: And as a-
Cathy Theys: And it changes too. We might have customers that will keep renewing their SOWs with us, but their needs change over time and so maybe they become a smaller contract. Or maybe they've got a big initiative and so they're growing the contract and we can adjust with that. That changes too.
Matt Kleve: As a support team you're growing too, right?
David Burns: We are. We are doing interviews right now to bring on a front end developer. We have a very strong support team right now, but nobody that has ownership of that front end expertise. We're a lot of full stack. We're really, really strong site builders, really strong back end developers and we can hobble our way through a lot of these front end requests. We get it done, we run tests to make sure everything's passing accessibility and delivering what the client needs, but we don't have somebody that we can just say, "This client's asking us about accessibility, this client wants to refactor the whole menu system on their site, go do it." So this new position we're hiring for, we're hoping to have that person onboarded maybe by April.
Matt Kleve: Very nice.
Mike Herchel: That's exciting.
Matt Kleve: That is exciting. What makes a support developer special? Are they just gluttons for punishment?
David Burns: Yes.
Matt Kleve: Okay.
Cathy Theys: I think it's like what we've been talking about, a little bit of it is attitude, like compassion and empathy, and understanding that decisions were made in the past. We're not building things from scratch, we're not making new decisions and we can consult and we can give advice if that's what's needed, but that's not the usual thing. And so I think you have to be okay with that. You have to be like I can't redesign this, I can't make this exactly follow my vision I have in my head. I'm working within the constraints of the system that it is. And it makes it easier, I don't know, mentally when you approach that from a place of wow, they're in a tough spot, they must have been in a tough spot before. It makes it easier to deal with that.
David Burns: I would say there's also a really strong communication because of the context switching, knowing what project's somebody's talking about and being very clear as to the type of thing that you're working on, because just saying the name of a project, may not be enough to have the person you're talking to fully understand and grasp, because they've been working on this other site for a bit, documenting things a bit more.
David Burns: And what Cathy said around the compassion. Clients are coming to us when something's broken and it'll sometimes feel like putting fires out all the time, but to understand that we're here to help them and we don't want to add any complexities to what they're already experiencing. So to take that, give solid estimates of how soon we could expect this thing to be fixed and always keeping the client up to date if our estimate is changing. I think client services does a good job of this, but they get to work in larger teams, with longer timelines, with bigger number of hours in a month. So being very cautious and aware of how we use the time and communicating that back to the client's like a skill set that our support team does very well.
Cathy Theys: Yeah, and I think being patient too, because we do see a lot of opportunities for improvements, but because we don't want to add any stress onto our customers, we have to be able to make a note of that and come back to it later, when the time is right for that. So you want to be looking for improvements, but also be very patient in terms of waiting for the right time for them, because we don't want to ignore those opportunities either.
Matt Kleve: Very good. David, anything else you'd like to add?
David Burns: I'm sure there'll be a link to the support and maintenance page on Lullabot.com, that has a bit more info-
Matt Kleve: Maybe.
David Burns: What's that?
Matt Kleve: Maybe.
David Burns: Maybe. And that will give a breakdown of some of the services we do and how to contact us if you need help. I should probably mention how our SOWs are structured. We like to work with a client for at least a quarter and then as we build trust in that relationship, we'll consider expanding an SOW to six months or even a year if they know they're just going to be sticking with this version of Drupal.
Matt Kleve: So it sounds like you wouldn't generally want to be like hey, I want you to fix my photo gallery this week?
David Burns: Yeah, this isn't like a jump in, jump out, because there's so much overhead with setting up the Lando, getting familiar with code base after like hey, we've got 30 hours this whole engagement that 15 hours of that is going to just be involved in conversations, gathering requirements. We really need three months to build our understanding and that relationship.
Matt Kleve: You probably feel like you're more useful to them at that point too, so being able to commit some time and not just fix the Drupal. Cathy, anything more you'd like to add as you point towards wrapping this up here?
Cathy Theys: No, I think support is great. We get to see so much variety of things and, I don't know, I feel like we really help our customers, and so it's interesting and rewarding.
Matt Kleve: So Mike-
Mike Herchel: Yes, Matt.
Matt Kleve: ... go support.
Mike Herchel: Yeah, go support.
Matt Kleve: And quite frankly, this may have come off as an infomercial, but I think there's a lot that could be gleaned by somebody who is a one person shop, say at a non-profit or something-
Mike Herchel: Oh wait, there's more.
Matt Kleve: ... maintaining a website, right?
David Burns: One more thing.
Mike Herchel: Yeah. For the low, low price of ...
Matt Kleve: But I think somebody who's-
Mike Herchel: No, I get it.
Matt Kleve: ... running their own website, might be able to glean a few things and think maybe if I change my processes this becomes a little easier, and automated testing or visual diffs, those tools could be very useful to them in keeping things going.
Mike Herchel: Absolutely. Yeah, absolutely. They're useful for everybody, not just folks in the support and maintenance department, but Drupal as a whole.
Matt Kleve: Three easy payments. 877-Lullabot. Call 877-Lullabot. That's our real phone number, by the way.
Mike Herchel: Yeah, I'm not surprised.
David Burns: It's funny, I have to always go to the website to actually see what the numbers are when somebody's like hold on, what number should I use, because I always go about it just typing the word Lullabot and have them figure it out on their own.
Matt Kleve: Well, thank you both.
David Burns: Thanks for having us. See y'all.
Matt Kleve: See you.
Mike Herchel: All right, thanks, bye.
Cathy Theys: Thanks.

Published in:

About the host

Matt Kleve

Matt Kleve has been a Drupal developer since 2007. His previous work in the media inspires him to create lean, easy-to-use workflow processes.

More about Matt

About the host

Mike Herchel

Mike is a former senior front-end developer at Lullabot, a lead of the Drupal 9 core "Olivero" theme initiative, and an organizer for Florida DrupalCamp.

More Resources

See all

Inside Lullabot's Support & Maintenance Department

Episode Guests

David Burns

Cathy Theys

Mentioned in this Episode

Transcript

More Resources

Module Monday: Content Type Groups

Module Monday: Pagepeeker

The Art of Estimation