Ease of Updates, Compounded Peace of Mind

Matt Kleve: March 14th, 2024. It's the Lullabot podcast.

Matt Kleve: Hey everybody, it's the Lullabot podcast episode 270. I'm Matt Kleve, a senior developer at Lullabot with me, co-host of the show Front End Dev, Morgan Eck. Hi, Morgan.

Morgan Eck: Hey, Matt. How are you doing today?

Matt Kleve: Outstanding. Glad you're here. So.

Morgan Eck: Likewise.

Matt Kleve: A lot of times when people build Drupal websites, they end up building a website, handing it off and leaving, and then that's somebody else's problem to deal with in the future. And I've been on a lot of projects that have been that way, but we're kind of on one now that's more long term, and we're doing lots of maintenance as well as building new hotness. Right?

Morgan Eck: Yes. Sorry, I was a little distracted. Yes.

Matt Kleve: Some projects that we work on have a lot of websites that all kind of need to be maintained at the same time.

Morgan Eck: Adding to that difficulty for sure.

Matt Kleve: And we have some folks that know what they're doing when it comes to that. First up, we have the VP of Extended support at Lullabot. He's starting with working with Drupal in 2005 and has worked with Drupal projects big and small, including Sony and Lifetime TV. He's the person who established the Lullabot support and maintenance department, and we'll be talking with a few members of that team today. Please welcome David Burns. Hi, David.

David Burns: Hey, thanks for having me back.

Morgan Eck: And we will also be talking to Cathy Theys today, who is our support team manager here at Lullabot. Cathy has been working with Drupal since 2006 and made many core contributions back in Drupal eight Big Push. She was also the first recipient of the Aaron Winboard award, Winborn award, excuse me, that's pretty cool. Hey, Cathy, how you doing today?

Cathy Theys: Hey, I'm doing super, glad to be here.

Matt Kleve: Also with us, we have a developer on the Lullabot support and maintenance team. He's been involved in Drupal since 2013. He authored Drupal eight for beginners. A student of Brazilian Jiu-Jitsu. Please welcome Jason Blanda. Hey, Jason.

Jason Blanda: Hey, it's great to be here.

Matt Kleve: Is Jiu-Jitsu a useful like Drupal skill?

Jason Blanda: It can be when you're frustrated.

Matt Kleve: I love it.

Jason Blanda: Good way to blow off some steam.

Morgan Eck: And we've also got Jenna Tollerson with us today. Jenna is the engineering lead for digital services Georgia. And it looks like that's a new title or group. And Jenna is an Acquia certified developer and has been building Drupal sites for over a decade. She writes and reviews code for Gov Hub, which is the state of Georgia's official web publishing system, and she also acts as a technical advisor for Digital Services Georgia Strategic Initiatives. That is a lot. Welcome, Jenna.

Jenna Tollerson: Hello. I'm so glad to be here with so many of my favorite nerds today.

Matt Kleve: It's been a while since we've been on a call together. Jenna. I was a part of the...

Jenna Tollerson: It's a pleasure to see you.

... initial team that kind of built out a lot of the the Georgia infrastructure. So if you if you don't mind jumping in first, tell us tell us about Georgia.gov and all of the Georgia sites.

Cathy Theys: Yeah. And tell us your new group title.

Jenna Tollerson: Our new group title is the Georgia Technology Authority Office of Digital Services and Solutions, which is at the state of Georgia.

Matt Kleve: That's a mouthful.

Morgan Eck: Wow.

Jenna Tollerson: It is a mouthful. Our main flagship product on my team is Georgia Gov Hub, which is a web publishing platform built on Drupal based on Drupal. That is a multi-site setup. So we host I think 85 sites now on a single code base where everybody has their own database. So it's georgia.gov. And then a lot of small agencies in the state of Georgia, and also a lot of big agencies like the Department of Revenue and the Department of Driver Services, which is kind of like Georgia's DMV, the Department of Driver Services.

Morgan Eck: Thanks.

Matt Kleve: So it's a multi-site setup, and every site has its own database. Is it all served off the same platform or is how does that work?

Jenna Tollerson: Yes, it's all served off the same platform. We do all of our everybody shares one code, code base. And then every two weeks we do a deployment and update all of the sites at one time in a big sweep, which is fun.

Matt Kleve: And I kind of mentioned that it's been a couple of years since we've last talked. So Georgia.gov is has been live with this platform and continually evolving over that time. Right?

Jenna Tollerson: Yes.

Matt Kleve: Something like that? I don't remember the timeline either. It's okay.

Jenna Tollerson: I don't... What is time? It is a flat circle. Yes. Georgia.gov has been continually, we actually, gov hub is the Drupal eight and up version of our web publishing platform. We came from Drupal seven and before that a proprietary system. So since, I think I think Gov hub itself was 2018 maybe.

Matt Kleve: Okay. That sounds about right.

Jenna Tollerson: And since and since then we've actually gone through a bunch of stuff. But the most recently we redid the entire look and feel with a brand new theme for all of Gov Hub called Georgia Bloom which we we have a botanical theme internally, so we name things.

Matt Kleve: So now that you have a few years managing the sites this way, you mentioned that you do updates on a regular basis. Is it still scary when you make those updates or are you comfortable with it?

Jenna Tollerson: Well, we have a lot of good processes in place and and I'm very comfy with it. And one of the things that's made it a lot easier recently is having a really good automated updater robot that keeps us up, that keeps the even like really tiny patch updates in place and keeps us moving.

David Burns: So so Georgia.gov, Gov Hub, was actually one of the first support clients when we started the department five years ago. And we've had the joy of, of learning a lot by doing the Drupal 8 to 9 upgrade which took many, many months.

Matt Kleve: Yeah, that one wasn't easy, but it was doable, right?

David Burns: Yeah, yeah. And a lot of it was 8 to 9 waiting for basically the contrib space to, to be ready. Right. So we probably waited six months, if not more after nine was released to actually start working on it. But but even to getting to nine took about six months and we learned a lot during that process. One of them being the benefits of automated updaters. So when we went from 9 to 10 that that timeline was drastically reduced.

Cathy Theys: Yeah. So what when you say we learned a lot about automatic updaters, what do you mean?

David Burns: So when we went from 8 to 9 we did most of it manually, right? We ran upgrade status and upgrade rector, which told us what modules were available to us with Drupal nine support and how much of our contrib custom modules and themes needed to be updated with removals of deprecations and new coding standards. And we wound up doing a majority of the module updates in, in groups like here's all the editorial modules. Let's create a pull request or release that that only focuses on the editorial modules, so we could point the QA team to just go test that area of the site. And we did those in phases and tried to get them out with each sprint deploy.

Cathy Theys: And those were manual.

David Burns: And those were all manual. Right? And we realized, like, we can't be spending six months every time there's a new core release. And around that time, you know, we started looking into things like violinist.io and GitHub renovate. And what was the other one?

Cathy Theys: Dependabot.

Matt Kleve: Dependabot.

David Burns: Dependabot. Yeah. Thank you.

Jason Blanda: Yeah. And I will say we did similar practices with 9 to 10 where we, we did a module inventory and we collected them into groups of like editorial modules or admin modules. And we ticketed all of that stuff. And by the time some of those tickets came along, the auto updater had already taken care of updating the versions of these modules to be D10 compatible. And so we were constantly updating our spreadsheets as well, because they took care of a lot of the work for us.

David Burns: Yeah, so if we look at Drupal 8 to 9 and we did our audit, we probably saw about like 75 out of 150 contrib modules that needed to be manually updated. And when we did that same audit going from 9 to 10, we probably had like maybe, I don't know, 35, 40 modules? I might be off by the numbers exactly, but it was significantly less work. And as Jason mentioned, by the time we got around to be like, oh, let's let's do this update. You know, renovate already created the pull request for us. And we we it we merged it in. We've, we've added automation around other projects that do like checks and visual diff checking. And if everything passes we actually auto merge on some of our projects, which is really interesting. I'm not sure if we've gotten to that point on Gov Hub yet.

Jenna Tollerson: On Gov Hub, we do auto merge any minor or patch update that has passed all of our automated testing.

David Burns: Oh that's great.

Jenna Tollerson: Yeah, which is really cool because then you wake up in the morning and overnight renovate bot has gone. I merged this for you. You're done.

Matt Kleve: For example, just to kind of lay it out there, if you have the A, B, C, D module and the contributor in Drupal made a dot one release of the A, B, C, D module renovate will run that or it will create a merge request, run the tests, and then possibly even auto merge if everything passes. Super handy. It sounds like having a good bank of tests is pretty important.

Jenna Tollerson: Oh yes, having we have a lot, on Gov Hub particularly we have a lot of test coverage that goes through and like clicks all the buttons and does all the things. And we have visual diffs with tugboat that needs to pass a certain threshold in order to get past it. So yeah, we do. We do have a lot of test coverage. So not everything gets to auto merge. But definitely especially for like really minor things, you're like, I barely even had to look at this. The robot knows what it's doing. So that's really nice.

David Burns: Yeah, it was great having Gov Hub handed to us from client services because it had a lot of those checks already in place. We found with the rest of our support clients that that don't have tests in place. The bare minimum of testing we want to do is doing the Crud operations on our entity forms. So like nodes, files, media, file uploads, documents. And the reason we've kind of like targeted that is, you know, I've been doing Drupal for way too long. I've noticed that when you install a module, if something is going to throw a fatal error or break your site it's usually in the editorial experience. And then for the front end stuff, we have our visual diffs that could catch, you know, like if a module is changing markup and, and breaks the page layout. So really no end to end testing being done for the user experience, just on the editorial experience.

Morgan Eck: Now, you mentioned Drupal 8 to 9 was kind of when we started bringing this in, was automated testing a thing in Drupal before Drupal eight nine, kind of transitioning? Like what made it possible around that time?

David Burns: So end to end tests, unit tests, like there has been automated tests in Drupal for quite a while. The reason it got important, at least for most of the people on this call, is because we started the support and maintenance department and we realized, like, if we're going to be adding value to some of our clients, some of whom are only purchasing like 30 hours a month or 60 hours a month. We can't be spending a majority of that time having a developer run composer commands do manual checking because that'll burn through those hours really quick. And when you do that, you tend to only prioritize like security releases instead of every single like patch release. And if you're only doing security releases, there may be a bug on your site. And when you investigate the bug, which might take like an hour or two hours, you realize, oh, here's a patch on the issue queue. Oh, this patch has been merged. And if we just updated this module, that bug wouldn't have been there. So by keeping your site up to date, you're, you're squashing things before you even realize that there may be issues. And most importantly, when you do get major Drupal version releases, like a majority of your code base is ready to do the major core bump.

Jason Blanda: Yeah, I think sometimes too like it... sometimes you'll grab a module that does everything that you want it to do, but it does it in a way that maybe you're not always thrilled about. There's just like this one little aspect that you're like, I wish they would fix that. I wish they would fix that. And if you ignore all of those minor updates, sometimes they sneak in that one little thing that makes this module do exactly what you want it to do, and you've spent who knows how long just wishing it was there and it was there all along. And if you're if you're staying up on these updates, sometimes you get the features that that you're hoping for.

Cathy Theys: Yeah.

Matt Kleve: What kind of tests do you run? People fight over that all the time like.

David Burns: So we started out with Cypress Tests and then our engineers, like, did a a project where they were using Playwright, which is owned by Microsoft. And, you know, we're like, oh, we're already started doing Cypress tests, we're just going to stick with this because we're familiar with it. But then being the data nerds that we are, we're like, well, let's actually write two very similar tests and see which one runs faster. And Playwright has been significantly faster. And we've also seen a shift in, in the communities where people were going from Cypress to playwright. I still think both technologies are great and use whatever one works best for you. Some tests are better than no test. But I think I see our team slowly shifting towards Playwright.

Jason Blanda: I agree. We also..

David Burns: And Behat. Sorry.

Jason Blanda: Yeah I agree and sometimes too, your testing framework needs to be evaluated. Especially if if your tests are a mess and they're all over the place. And sometimes we like to write tests just to write tests. And that can significantly slow down the process of these auto updaters. As we're waiting on things and then those tests for tests sake can break, and then you go to fix it and you're like, why does this exist at all? And so sometimes in those situations when you realize you have a lot of bulk, that's when you should maybe reevaluate if we're going to have to rewrite all of these tests anyways, do we want to switch frameworks or do we want to stick where we're at? But to Dave's point, having tests is better than not. And in our department specifically, if you're coming to us with tests already in place, yes, we're going to evaluate those for quality and usefulness. But, you know, I think sometimes we got to be versatile enough to to know the testing frameworks that are out there, whether it's Cypress, Behat, Playwright and work within those. I think the goal more is, are these tests useful and can they give us an added layer of confidence when this auto updater just updates a bunch of stuff? Can we trust that our tests tested the things that are important? And when this auto merges, we can have a high level of confidence that is not going to break anything.

Cathy Theys: Jenna, why don't you tell us a little bit about Gov Hubs test runner? What, like, what's it using right now? Kind of, you know, give folks an insight there.

Jenna Tollerson: What's it using right now? Well, right now we have a lot of Behat tests and also PHP unit test and Linters and all the other good stuff. But we're actually probably looking to retire our Behat tests and move over to Playwright. Not just because it's faster, although it sounds like it is much faster. We're excited about that. But because it also more modern and maybe a little easier to write the tests. We don't have as many people who are not developers writing tests, so being able to put things in Gherkin format is not necessarily like not as necessary for us as it might be for other teams, so.

Matt Kleve: Isn't Behat the one that is the plain text, like the actual test file that gets written as plain text?

Jenna Tollerson: Right. So like given I am an administrator, when I click this thing, this thing should happen more or less. And the the idea is, is, is somebody who's not a developer can write a really cool tester on that, but, but instead just sort of adds an extra layer for us in particular.

Matt Kleve: There's also code that has to back up the "I am administrator", which...

Jenna Tollerson: Yes.

Matt Kleve: ... you know, is the code that...

Jenna Tollerson: That does have to be written by a developer. Yeah. So so it breaks it apart into two separate pieces. Yeah.

David Burns: Yeah. Where like Cypress and playwright, they're JavaScript based. So you're doing like DOM selection and your, your checking values put into fields. And it's really neat, it's modern, it's quick. And we found that like, you know, if if a developer is able to write JavaScript, they can write test, which is most everybody on the development team.

Jason Blanda: Yeah. And I don't even think you need to be like a, a high level JavaScript developer to be able to understand these tests. If you can understand how JavaScript is formatted and even just look at existing tests with any kind of, you know, knowledge level, I think you could get in there and start writing any kind of automated tests using these JavaScript forward frameworks so that the barrier barrier to entry is kind of low, which is really awesome.

Matt Kleve: Plus both...

Cathy Theys: Especially when there's already like 1 or 2 tests. Those are the hard ones. And then once they're there, copying and pasting and adjusting selectors and actions. Yeah, I agree with Jason. You don't have to be a JavaScript expert to to to do that. I think I could even do it.

David Burns: You could totally do it, Cathy. What's cool is both of these have plugins that you could actually record your actions in the browser and then export what you've done into a test. All that JavaScript, and then you could tweak it, you know, clean it up, make your selectors better. Because it tends to be a little more verbose when you're clicking through on like, how it's doing the selectors for field names and stuff.

Jason Blanda: Yeah. And caveat to that, they never work as well as the demo video showing how to do it. There's always hiccups along the way. So you likely are going to have to go back in and modify some stuff.

Matt Kleve: Jason, you know that when you make a demo video, it's it's a very specific path that you're recording and you know, it's not going to go wrong because you've done it 17 times because it broke 16 times before that. Right? So of course, the demo video looks great.

Jason Blanda: Oh, absolutely. You should see the other demo videos before that demo video.

Matt Kleve: [Laughter] Oh crap, I forgot that. Never mind. Let's try again. We're talking with the Lullabot support and maintenance team and Jenna Tollerson from Georgia's state government, running a bunch of websites on Drupal, figuring out how to keep them all up to date and deployments running smooth. Coming up, we'll have more on the Lullabot podcast right after this.

[Intro music]

Morgan Eck: Welcome back to the Lullabot podcast. We are having a chat with a few members from our Lullabot support and maintenance team, as well as Jenna Tollerson with us today, and we're talking about managing software updates for hundreds of websites. Welcome back.

Matt Kleve: Jenna. Now you manage software updates for like 80 some websites. When, when did you start Drupal? How did you get involved with Drupal? Did you ever think you'd be doing 80 some websites when you installed Drupal for the first time?

Jenna Tollerson: I got involved with Drupal, it was I don't I don't know how many years it's been that it was it was it was a long time ago. But like a lot of people, I was building little websites for, you know, specific clients and like hand rolling CMS's so they could make updates to their own websites themselves, which as we all know, is a huge pain and probably not using the best security practices either. While I was at it. And one day I was like, there's got to be a better way to do this. And I did try out a few other CMS's which I will not name. And then one day I installed Drupal and was able to put together all of my forms with, y'all might remember this, the CCK is it CCK or.

Matt Kleve: Oh yeah! CCK otherwise known as Drupal now, right?

Jenna Tollerson: Yeah, right. Otherwise known as Drupal now.

Matt Kleve: The CCK was the content construction kit. It was the early like put fields on a node thing. So if you want an image on a node, you put in an image field. If you want a text area, you put in a text area and you can really create structured data. That was Drupal's early win in my opinion, right? CCK.

Jenna Tollerson: It was a huge win though, because I was like, oh, I could just build this form and then I can build a bunch of views, which is also now just part of Drupal, and then just focus on like making a nice looking website for like this small business that sells soap or whatever, which is a thing that I was doing.

Matt Kleve: Were you doing Uber, Uber cart or commerce or anything like that? You were saying you sold soap.

Jenna Tollerson: So yeah, I did. I did use Uber cart. I had forgotten what the name of it was, but it was Uber cart. That's right.

Cathy Theys: Oh my gosh. Back in the day.

Jenna Tollerson: Back in the day. Yeah. So if you're if you're if you understand how time works and you're keeping track, you can get an idea of how long I've been doing this. And then almost 13 years ago the state of Georgia was moving from a proprietary content management system to Drupal seven, and they needed somebody to work with the vendor they're working with at the time and like, translate all of the nerd language, more or less. They wanted to bring someone internal to the team to be their own nerd, their own nerd that could interface with the other nerds. And that was me. And I was brought on just to help with this migration. And then they were like, we feel like you're doing really good stuff here. Maybe you should hang out. And now it's been over a dozen years later and a couple different builds of Drupal.

Jason Blanda: I will say, they were they were absolutely correct in the decision to allow you to continue hanging out.

Jenna Tollerson: Well, thank you so much. I really appreciate that.

Morgan Eck: So I know in another life I was a support developer before I was a front end developer, and towards the end of my life doing that, it would take me quite literally all of my time to do updates. And so before this infrastructure was in place, how long would you say it would take a member of the team to complete all of these updates?

Jason Blanda: I know for me, from past projects, not necessarily this one, but when I was doing, like Jenna was saying, managing for for small businesses and things like that, I might only have 5 or 6 websites that need module updates and core updates, and that is that is a day's worth of work between FTPing files, because you didn't have composer and doing the updates locally and and all that fun stuff. I mean, you were you were talking not just a day, as in like, oh, I'm doing this for today. Like a full eight plus hours, just updating modules and core.

David Burns: I'm sure there's people listening right now that are just like, just run composer, update and and get all the updates at once and create a pull request for that and everything's good. Yes, that that would probably work for a majority of sites out there. But knowing that you've got 85 plus websites running on a single platform you want to be very specific about your testing. And let's say you did run composer update in a single PR and it does fail. What what module caused that to fail? If you got 20 things being updated at once, having this automated updater in place, which creates a single PR for each package that needs to be updated. If a test fails, you're like, oh, this is the thing I need to look at. It helps you narrow your scope on fixing things. And I see that as a huge benefit. So if you were to need to update, say, even five modules a week, and you need to do QA testing, regression testing manually, like at on average, say two hours per module which would be ten hours a week 40 hours a month at least. That's a lot of time, right?

Jason Blanda: Yeah. And I think we we didn't mention too, that we're talking about this auto merge thing. We're not auto merging into the main branch. All of these, like Dave said, all of these auto updates per module will create their own pull request. But when all that stuff passes, it merges into like a, a branch for that sprint automatically. And then we have another chance to QA run tests on this branch that has all of these updates together. So we're testing each PR individually and then we're testing them again as a group. And then that ends up being merged into the master branch. So we're not just like willy nilly, "oh, this one passed, let's throw it into the master branch!" There's like a middle ground before it all gets there.

Cathy Theys: Yeah. My couple projects that I PM also use an intermediate branch for updates. On one customer, we do that because the customer has their own internal development team, and so they're pushing to master and if we were pushing all of our Renovate updates to master, they would be updating their PRs, like, all the time from these auto merges. And so we still get the test coverage at the individual module level, which reduces our time to investigate when something goes wrong. But the. In intermediate update branch contains our noise so that it doesn't bother other active development. And let me think my I think my other project kind of does it for similar reasons, but it's not as active a project over there. So it's kind of an option for that project to merge into master. I think it's really project dependent. Dave, when do you think merging the automatic updates with the auto merge to master, like when is a project a good fit for that?

David Burns: Yeah, if a project is stable and there's really no active development going on, I believe one of the other projects you were referring to, there's actually another agency working with them?

Cathy Theys: Yeah.

David Burns: So we don't want anything that we're doing as the support team to create any blockers for them, because time is money. So having this integration branch that the client gets to decide, hey, we're ready to QA all this stuff now we're going to merge it in. Actually, we're going to test it in its own branch, make sure everything's good there, which we have automated tests doing as well. And then we're also going it's that branch is being rebased with Master or Main. So it's almost an exact copy. As if they did merge that stuff in. There's a high level of confidence throughout this whole process that whatever is being merged is very unlikely to be introducing regressions.

Cathy Theys: Yeah. Yeah. And like. Oh, so some of those high level of that high level of confidence comes in part not just from like the end to end or the PHP unit tests and the linters that we've talked about. But there's also another suite of automatic testing that we aim to have. Like, check that there's no overwritten config. Were there update hooks run. We've got a schema check that is a Lullabot tool that we wrote so that we know when update hooks are run or not.

Matt Kleve: Is that is that public? I'm familiar with the tool. I didn't know if we had that in our public repo.

Cathy Theys: I don't know that it's public.

David Burns: So we have, there is a public schema checker.

Cathy Theys: That we use! We're using the public schema checker. It was mglaman?

David Burns: I believe that's...

Matt Kleve: Glaman yeah.

David Burns: Yeah.

Cathy Theys: Yeah. So so we've got those checks. We're also automating like, the check for the status page doesn't have errors. And we're automating a check that there's no new errors in the revision the recent log messages. So I guess I'm trying to emphasize that this auto merging, this high confidence that we have from this auto merging has resulted from a lot of investment that we've made into automating what people might manually do as peer review. And then because Lullabot has this kind of like standardization initiative that we've been working on for a few years that's coming to fruition in the maturity of the Drainpipe project, which is a public open source project where we're putting this work that we've had to do on individual customers to do this automation. And we're, as we're have done it over and over in different situations, we start to realize, well, what's generic enough that we can reuse? And then we're putting that in drainpipe. So when we get a new customer now we can get these automated tests in place a lot faster than we were doing three years ago.

David Burns: Yeah. So drainpipe is like a way that we could do continuous integration and add that to a project in like significantly less time than if you're, like, rolling it out specifically on each project, like piece by piece.

Cathy Theys: And we want other companies to be using it too. It's like the issue queue is open. You can make pull requests to it.

Matt Kleve: I think there was a Drainpipe channel in Drupal Slack that I saw. Somebody was like actually using it, and there was kind of some surprise. It was like a oh, I think you're the first ones now. Cool. Glad you're here.

David Burns: There's a check in there that isn't common. People overlook even as they're doing QA is like, run cron. Like, there's been so many times in the past where we test everything and it's perfect. We push something out, and then people were like, hold on, how come our importer stopped working? Or how come, you know something's not refreshing? That should be happening in cron. And we run cron manually. And we see some like PHP error and we're like, oh. So part of Drainpipe and part of our workflow and checks is confirming that cron can be running no matter what we push.

Matt Kleve: One thing that I kind of just spiked in my mind, and I know that we got to rewind a couple of minutes to to where we were talking about it. You were talking about using Renovate to update modules and the whatnot. And I think without saying it, like the conflicts, the biggest conflict that I see out of that is composer.log files. So is that what you're trying to avoid when you're using that secondary branch, that merge branch before you merge it back in?

Jason Blanda: It doesn't matter what your target branch is, whether it is master or it is some intermediate branch. When that target branch changes, say something auto merges in any open PR that maybe the tests are still running or there's something going on, will rebase themselves with the changes in the target branch.

David Burns: And generate a new hash for that composer.json file or composer.lock file.

Matt Kleve: Even if it's my manual adding a module pull request?

Cathy Theys: No, no. So that's where the like I think getting back to the integration branch comes in. So like additional features, non-maintenance things that may have composer changes that go into master your automatic updater, if it's targeting an integration branch can keep running and then you only I'm going to say, only need to resolve the conflicts between your main branch and your integration branch. And sometimes that's super easy. It's as easy as picking a composer.lock content hash. And it doesn't matter which one you pick. And running "composer update --lock" to give you a new hash. Those are like my favorite conflicts to resolve because it takes me two minutes. But when you have more complex changes especially when the feature that's going into the main branch is changing the same things that are getting updated. That's like not usually adding a new module. Adding a new module is not that too complicated. But when you're touching the same things and then then that can take sometimes some hours.

David Burns: For example, like a contrib module may introduce new configuration. And that's one of the checks that we do which would fail. And that's usually when a developer would step into that branch and just do config export. So it's captured and that'll go into the integration branch. But if anything's been done in the main branch that touches that same file, that's when you would need to like resolve those merge conflicts manually.

Cathy Theys: Yeah. So that's that's one of the benefits to like if you have a low velocity project merging directly to main can remove that complexity from the system. But it's really project specific.

Jason Blanda: Yeah. And one of the things that we're doing actually on the Georgia project, and this is something that Cathy had suggested you know, we're working in two week sprints and Cathy's like, hey, why don't we why don't we do the composer and the NPM package updates like early on so that all the other tickets that could influence and change some of these things already have all these updates, and then it kind of gets out of your way. So we've been trying to push that earlier on so that we don't run into those conflicts by the end of the sprint, where we're trying to merge in, say, eight module updates and there's a ton of conflicts. But we did it early on, and now we don't have to worry. And that's the other thing. We're not just using these automated updaters for modules. This is this is also updating, you know, npm packages that the theme relies on as well.

Cathy Theys: Yeah.

Jason Blanda: So the Renovate at least not I don't think all of them do. But Renovate at least will do composer updates as well as package.json updates. So we're kind of leveraging both halves to to do all the things that we need to do.

David Burns: I've even seen it at times updating config files where you're loading like Docker images that'll be like this is the preferred Docker image and it'll create a PR for that.

Morgan Eck: I know we have an article about this, so we'll link to that in the show notes. But do you have any like must have settings or preferences that you apply to all of these projects when you're setting them up? You know, all of this automated infrastructure.

Cathy Theys: Well, let me jump in first here because it the thing I have in mind relates to something we were talking about earlier, where we were saying that we were doing all the module updates separately, which in broad strokes is true. But there are cases where we customize our, our Renovate config, for example to group packages together. So for example Drupal core, Drupal core recommended, Drupal something I forget there's like three.

David Burns: Project Messages.

Cathy Theys: Project Messages.

David Burns: And then Scaffold.

Cathy Theys: Yes! And so we have those all grouped together. And then there's also like some projects where I've grouped all the jquery stuff together. And, and I think that that configuration allows a nice balance between, like, strictly every single dependency and doing it all at once. Like you can actually configure it to group it how it works for your project.

Cathy Theys: What other like favorite configs do people have.

David Burns: It would be a little wonky to have, like the Drupal core version going up to like the next version of Drupal Core, but then your scaffolding is still stuck on the previous version. So that's definitely a reason to be like, no, we definitely want all three of these to be updated at the same time. I've also seen like web form modules and other extensions of web form modules needing to be updated together because they usually are go tandem. What other ones have I seen? We're grouping basically all the jQuery packages together and all the gulp packages together. Instead of doing those individually.

Jenna Tollerson: We group meta tag and schema meta tag together in GovHub because they so often just go right together.

Matt Kleve: Makes good sense.

Cathy Theys: Yeah and and then there's other config options like, Oh here's a good configuration. So Jenna was saying earlier how she's configured auto merge for the patch and minor updates. And initially on my projects, I had configured major versions to not even give me those. But after running Renovate for a while and getting all caught up with all the minors. Then I enabled the major versions to come in, but only with approval. So we still have to like as a developer or project manager, go to the well for Renovate, it's a dependency dashboard and you can check a check box and say, I'm ready to try this major version update from version three to version four. So it'll it'll still do it for me, for us to, like, get that initial step. But we can control when it's doing that, when we're ready to work on it. Because major versions, we're going to anticipate some manual some manual work.

David Burns: Actually pulled up a configuration file to find some of our other things that we use. So we do branch prefixing. So anything that Renovates correct creates we do renovate forward slash. And then whatever the name of the package is. We do a concurrent limit for the number of PRs that are created.

Cathy Theys: Oh, that's really important.

David Burns: If you don't put a limit, it's going to be like, oh, you got 20 packages to be updated? Here's 20 PRs, right. And if you've got automated test kicking off, that means you got 20 automated tests running. Well, however many tests per pull request being triggered at the same time. And then when one of those gets merged in, it needs to rebase all the other ones. So then you're kicking off the same number of tests again.

Speaker8: Yeah, then it's like 19 and then it's like 18 and it can use up all your resources, your GitHub minutes, it can make things time out.

David Burns: Yeah so we put a limit on that to like 2 or 3. And what we found is sometimes we'll have flaky fails. You know maybe the page didn't load quick enough. Right. So it's like, oh, this PR is not going to work. But when it rebases, the next time, everything runs great and everything passes. So yeah, we put our current limit to 2 or 3, and then we also have an auto merge schedule set for some projects. So you could say like, I want to do this outside of business hours. So after 5 p.m. until like 8 a.m. in the morning, you can have Renovate running instead of all day long. Or you could just say like only run on the weekends. It's it's pretty awesome.

Jason Blanda: Didn't we, on one project run into an issue where renovate was trying to create PRs for dependencies of dependencies?

Cathy Theys: Oh, I think there was like a package.json inside a theme. And then there was also a package.json in the main in the docroot. Is that a situation? The situation you might be thinking of?

David Burns: No. Like child dependencies, right? So if. I can't even think of a contrib module that has multiple now, but let's say webform had.

Cathy Theys: Webform, right?

David Burns: Packages that that are needed. It'll actually create a pull request for the sub-dependencies. And you may not necessarily want that, because if you update those, they may not be compatible with the version that webform is expecting to be there. So I forget what setting we have that that turns that off. But yeah, you probably don't want to do sub-dependencies.

Jason Blanda: So that alongside not limiting your PR numbers, you could find yourself with an extreme amount of PR overload and might panic a little. And the other thing with these, with these PRs is to, you know, if you're not auto merging, we still have the ability to say, no, don't do that. You know, not not everything bypasses us entirely. If it's if it's an update we're not ready to commit to or something we know is broken and it we can we can say, hey, no, hold off on this.

Matt Kleve: I for one, welcome our robot overlords. Somebody who may not be familiar with any of these tools. Their head might be swimming over the last, you know, few minutes because we're we're talking in vocabularies that are probably fairly unfamiliar. Jenna, you've seen this grow on your websites over time. How do you feel about it? Is it is it a welcome addition? Is it a bunch of work? Is it useful overall? It's it's helpful, I'm sure.

Jenna Tollerson: It's very helpful. I, I wrote most of the Renovate config we have running on GovHub and a couple other repos myself, and there was a lot of iterations to get it, like to watch it run and then get it tweaked exactly what we wanted. But the, the documentation for Renovate is really, really good. Outlines, all the configuration options. So many configuration options. But it is all there and very descriptive. And then you just run it and and go like, is that what I wanted? There's also when you when you turn on initially there's an onboarding PR it automatically creates an onboarding PR that's like here's what we suggest for your project. So like to get you started and to tell you what's going to happen based on what is in that branch. Once you actually merge it.

Matt Kleve: It sounds like it's a lot of work, but in the end, that work ends with some time savings and some benefits for the maintenance.

Jenna Tollerson: Absolutely.

David Burns: It's almost like adding a whole new developer to your team.

Jenna Tollerson: It is so much like adding a whole new developer to your team. Yes. And we have we have a renovate bot user with a little robot face that just comes up all day saying, here's the things that I did for you.

Matt Kleve: I think renovate should look like a kitten, but that's just me.

Morgan Eck: Aw!

Matt Kleve: Right?

Morgan Eck: Everything should look like kittens.

Matt Kleve: Do you have anything to add Jenna, before we point toward wrapping this up?

Jenna Tollerson: I was thinking about when you're talking about integration branches, how I've actually set up a pipeline. So if we update composer.json or update package.json in master it automatically syncs it back to the integration branch. And also every time we cut a release tag, it automatically syncs it back to the integration branch, the integration. So the integration branch or the the updates branch is like never too far behind the the rest of what we're doing. And if for some reason that that merge fails we get a notification about that and somebody can go like, fix it for real. But most of the time it doesn't fail, which is pretty cool.

Matt Kleve: That's great. Jason, do you have any final thoughts?

Jason Blanda: To be honest with you, life before these auto updaters kind of sucked and some genius decided to put composer into Drupal, and then some genius decided to leverage that composer.json file and create these bots that are whole new employees for you. And to be honest with you, as we navigate through our weekly sprints and things like that. It's nice not to be like, well, did somebody check and see if we had any updates? And our PR is created and, you know, did we is that ticket in the sprint just to have it and know that it's done. And even though occasionally it requires some human interaction, for the most part it does the tedious stuff while you get to do the fun stuff. So I love it.

Matt Kleve: Yeah, the whole like, hey, reminder, there was an update to this module that you haven't thought about in a year. Yeah, that's that's a great thing.

Jenna Tollerson: To me, it's such a huge savings to not be like, did somebody check to see if there's available updates? Just that step.

Matt Kleve: Yeah.

Jenna Tollerson: Nobody actually having to do that is, is just great all on its own, even if you don't do anything else.

Matt Kleve: Cathy?

Cathy Theys: Oh, thanks for picking me before, Dave. So I can channel my inner Dave. Just the more checks that we can automate a base level of crud for end to end tests and some config status checks. I just the more checks that we can automate, just the nicer our lives are.

Matt Kleve: David?

David Burns: Yeah. Cathy's pretty much nailed it. Like this forced us.

Matt Kleve: She channeled Dave! She did it!

Morgan Eck: She did it!

David Burns: It forced us to put automated test in place, Whether that be end to end or you know, other checks that are happening. So we could get to the point where we're doing auto merging and kind of just forgetting, not forgetting, but, you know, getting it out of the way. Because I really think one of the, the biggest values that we've created as a team is my mantra of like, automate maintenance so we could humanize support, like, let's be in slack talking to our clients about like, the things that they're struggling with as editors of the site, or users of the site or developers on the site. Don't worry about the maintenance like there's little value. As a developer running composer update, you know, nine times a day, like, let the machines do that.

Matt Kleve: Very good. Morgan, I'm glad we have people like this who have thought about these things and can help integrate them into the lullabot projects we're on.

Morgan Eck: It's amazing. Yeah. I'm sure this will be coming to us soon.

Matt Kleve: Thanks, everybody, for joining us.

David Burns: Thanks for having us.

Morgan Eck: Yeah, thanks, everyone.

Cathy Theys: Thanks!

Matt Kleve: See ya.

Jenna Tollerson: Bye.

Jason Blanda: Bye.

Cathy Theys: Bye.

Matt Kleve: Thank you.