The End of the Road for Drupal 8, Many More Drupal 9 Beginnings

Matt Kleve: September 16th, 2021. It's the Lullabot podcast.

[Intro music]

Matt Kleve: Hey everybody, it's the Lullabot podcast, episode 255. I'm Matt Kleve, a senior developer at Lullabot. With me, as always, co-host of the show senior front end developer, Mike Herchel. Hi, Mike.

Mike Herchel: Hey, Matt. Happy Friday.

Matt Kleve: Happy whatever day this release is. Hopefully Thursday, right?

Mike Herchel: [Laughter] Yeah.

Matt Kleve: That's usually been our goal. And you know what, Mike? Do you know when we started doing this podcast?

Mike Herchel: No.

Matt Kleve: You and I hijacked the podcast Thursday, November 9th, 2015. And you know why? Do you know why that date's important?

Mike Herchel: No. Why?

Matt Kleve: Because it was the day that Drupal eight was released.

Mike Herchel: Yes. Oh, yeah, I do remember that.

Matt Kleve: Remember, we wanted to kind of coordinate that. We kicked off the podcast with a new version of Drupal, and Drupal eight has been in the wild since then, right?

Mike Herchel: Yeah!

Matt Kleve: As long as you've been running wild on the podcast.

Mike Herchel: That's a long time.

Matt Kleve: It is a long time, and Drupal eight is coming to an end. And that's why we're here today.

Mike Herchel: Yep.

Matt Kleve: We're going to talk about the end of life slated for Drupal eight coming up this November, November 21. And...

Mike Herchel: Yep!

Matt Kleve: ... people need to be ready for that.

Mike Herchel: Yeah. And with us, we have a couple guests we have. First up, we have our support project manager from Dulles, Virginia. Cathy Theys. Hey, Cathy.

Cathy Theys: Hey there everybody. Glad to be here.

Mike Herchel: You were just on the last podcast.

Cathy Theys: Oh, yeah. I got a bug up my butt. I was like, I got things to talk about.

[Laughter]

Cathy Theys: We need to talk about some things. And I scheduled them, and here we are.

Matt Kleve: And sometimes it's bugs that make sure that these things happen. So thanks for that. We appreciate you being here. Also with us from the Support and Maintenance department, we have the Director of Support and Maintenance from West Philadelphia. Joining us is David Burns. Hey, David.

Mike Herchel: Born and raised.

David Burns: Hey, y'all glad to be here. This is actually my second podcast this year. And for somebody who rarely does podcasts, that's like a big milestone for me. So thanks for having me back.

Mike Herchel: Thanks. And next up we have a lead engineer from Minneapolis, Minnesota, doctor Matthew Tift. Welcome, Matthew.

Matthew Tift: Well hello everybody. Glad to be here. I have not been here since April 2017.

Matt Kleve: Now that we have all of our dates figured out, you may know Matthew from the Hacking Culture podcast.

Matthew Tift: That's true. You might.

Mike Herchel: Or from lots of articles on what were you talking about? You're talking about, like, the credit system for Drupal core. You've been working on a lot of that type of stuff, too.

Matthew Tift: That is correct.

Mike Herchel: That could be a whole podcast, by the way. Gotta write that one down.

Matthew Tift: It could be a whole podcast.

David Burns: I'd listen to it.

Matthew Tift: That's a great idea, actually.

Mike Herchel: Yeah.

Matthew Tift: But I've been spending a lot of time with Tim Lennon and trying to help port or help work with the folks at GitLab to port Drupal's contribution system over to Gitlab, as well as other potential places where people coordinate on software like GitHub and elsewhere. But right now we're focusing on GitLab, so.

Mike Herchel: We're not talking about that in this podcast. This podcast we're talking about Drupal eight, end of life.

Matthew Tift: That's right.

Matt Kleve: Drupal eight is coming to an end. And what's been advertised is, hey, if you've upgraded Drupal before, this is a totally different experience. And I think for some extent that's true, right? But with us is some folks who have been through the process. I was on a client project that recently did an 8 to 9 port and it was fairly successful. I don't think it's quite launched yet, but I've rolled off the project sadly. Cathy, maybe you want to. You're the one that said you had the bug up your butt about this and you wanted to, you know, talk. What? What do people need to know? Like Drupal eight? Drupal nine?

Cathy Theys: Yeah. So I think we've got a lot of people who know about Drupal or have Drupal sites have heard about Drupal nine, and they know that it's out there. But I think a bunch of people are making like, assuming that Drupal eight will have some kind of extended life and extended support to help, you know, bridge the gap and deal with all the things. And that is not the case for Drupal eight. It is for Drupal seven. Drupal seven is going to be around longer than Drupal eight. So if you're on Drupal seven, you're kind of okay for a little while, but if you're on Drupal eight, you really need to move to nine and you gotta do it real soon. And I think that difference is going to surprise a lot of folks.

Matt Kleve: It sounds like a surprise, but this is something that I mean, that date has been set for quite a while now. It's just been if you haven't heard about it, sorry to be the bearer of bad news. Let's let's get this going. And there's a there's a good series of articles on on Lullabot.com, and we'll link to those in our show notes. You know all about Drupal nine and what you need to know, but I suppose this podcast here is kind of the the short version, right?

Cathy Theys: Yeah, yeah. So we just I just really want to make sure that, you know, we use all the media that we can. So we've got a lot of good articles that have been coming out, you know, over the last year about this. And, and I was like, let's use another media stream, let's use the podcast and get the message out that way too.

Matt Kleve: So, Matthew. As a developer and somebody who's gone through the upgrade process, to you, what makes Drupal eight different than or I'm sorry, Drupal eight to Drupal nine different than Drupal eight to Drupal, Drupal eight from Drupal seven. So 7 to 8 versus 8 to 9, I guess is the question? If I can get that out clearly.

Matthew Tift: Yeah, that's a good question.

Matt Kleve: Well, I don't know if it was. It was worded quite right, but it was, you know, that's the point of the question. [Laughter]

Matthew Tift: I think you did a fabulous job, Matt. And I know exactly what you're asking, because I, in a way, am here because I have heard over the years about how difficult it is to upgrade from, say, Drupal 6 to 7 or 7 to 8. And once we got to eight, a lot of people have been saying, oh, it's it'll be so much easier to go from 8 to 9. And I have heard that quite a bit ever since before we even had Drupal eight. And then what I have found surprising on so many projects is that that upgrade from Drupal 8 to 9 was more tricky than I thought, and I encountered all sorts of new problems that I didn't have really a lot of experience dealing with. So I would say that there is a very different set of problems that people encounter with Drupal 8 to 9. A lot of them have to do with composer and composer conflicts and things along those lines. But in general, I have found even both on, let's say, a personal site like say, you know, I've been wanting to upgrade my own personal site MatthewTift.com from Drupal eight to Drupal nine, but the markdown module isn't ready. And I love writing in markdown, but I've just been kind of sitting and waiting and contributing a little bit, but just kind of watching an issue to wait for that to work.

Matthew Tift: But there are things like that where either I need to jump in and do something or I need to find I need to be okay with the hacky way, or I need to wait for somebody else to do something so I can do it as easy as possible with as little effort as possible. So that's kind of the one end where maybe there's just one module that kind of holds you, holds you back, and then on the other end, working for clients. I've found some extremely complex problems that have come up because of, say, custom code or people who have been trying to solve composer conflicts. And composer told them to do something, and then they locked themselves into a version. And then all these things where I have just encountered a weird host of problems. But in all honesty, I haven't really done a lot of successful. I've done like two successful Drupal 8 to 9 upgrades, but I've done a lot where I'm still in the process. And that's that's that would be my big message initially is that from little sites to big sites, there might be one thing or lots of things that could trip you up, and that the process isn't necessarily as easy as we might want it to be, even though it actually isn't all that difficult.

Matt Kleve: So if you have a Drupal eight site and you haven't looked at it yet, you need to look at it yesterday.

David Burns: Yes.

Matt Kleve: And really start to think about this. David, I noticed you were smiling and nodding when Matthew was talking about composer issues and that kind of thing and changes that might not be apparent from a code perspective.

David Burns: Yeah. So the the way I remember Drupal eight to Drupal nine being pitched was it should be very similar to a minor point release or, you know, an 8.8 to an 8.9 release. And I think in some instances that is true. And we could go into more detail about that a little later. But to speak to what Matt Tift was, Matthew Tift was talking about we've updated three sites and currently have like two more sites in progress going through the upgrade process and everything that he mentioned about, you know, just composer really getting into the way and people making decisions to pin specific modules adds a lot of complexity and things that people probably had not accounted for in terms of doing the upgrade or even just running the few modules that give you insight as to what you need to upgrade and how it needs to be upgraded. And by that I mean the Upgrade Status and Upgrade Rector modules.

Matt Kleve: Yeah, let's talk about that. I was just looking for the name while you were talking, David. So the Upgrade Rector module is a tool that exists. So from the, you know, the most widest perspective, a lot of the changes from 8 to 9 are code changes. There were some deprecated APIs early in the eight process. That won't work at all in nine. And you need to make those changes to your code. But there are some cool tools out there, like the Upgrade Rector module that help kind of automate that process a little bit, or at least point out that, hey, dummy, this doesn't work anymore, find a better way to do it.

David Burns: Yeah. So Palantir wrote something that integrates with PHPStan basically scans your entire code base. You can specify what folders it should look into and says, hey, we noticed this is not going to be compatible with Drupal nine. It even goes as far to say this is the line of code that has changed, here's what we suggest you replace it with, and here's the page on Drupal.org where you can read more about this issue, which is fantastic. It goes even a step further that you can actually execute Rector on those modules, and if it's able to make those changes, it'll do that for you. And we've noticed by doing that, that gets you pretty darn close to being D9 compatible with these modules.

Matt Kleve: Yeah.

Mike Herchel: I have a question right here. So like I'm hearing the Upgrade Rector module, but there's also an Upgrade Status module. What is the difference between those two modules?

David Burns: Yeah, they work really well together. So Upgrade Status will basically, it's kind of like the Update module in Drupal core, right? It kind of looks around on Drupal.org and sees if there's a newer version, it kind of takes that functionality and extends it a bit more and says, yes, there is an update available. And is that update compatible with Drupal nine? Yes or no? And then modules can actually plug into that like Upgrade Rector. That includes the information I told you before about replace this line of code, it has one warning and three problems and tells you what code to replace. So those two modules work together in a suite to give you a very clear insight as to the amount of effort it's going to take to get your site onto D9.

Mike Herchel: Would you consider that as like the first step? Like if you have a D8 site, you don't know what the heck's going on, i'm just going to plug in these modules and see what happens.

David Burns: Absolutely.

Matt Kleve: 100%. Yeah. I agree.

Matthew Tift: I have a comment on that as well because I thought, all right, I'm going to, for my client, go, you know, install these modules and get a sense of what I need to do and come and report back and say, here's all the work we have to do to upgrade to Drupal nine. And then I spent like a week trying to get the Drupal Upgrade Status module working on the site, and part of that was me thinking, well, I had also heard, in addition to Drupal Rector, I've heard about Drupal Check. So if you go to the Drupal Upgrade Status module, it'll it even provides a nice little checklist on the home page of Drupal Check does this and Drupal Upgrade Status does this. And then there's also Drupal Rector, which we've talked about. And I thought, well, Upgrade Status module has got a lot more check marks. I want I want that one. And then I kept having issues with it. And then that module also says, well you might have some issues where like you need to remove Drush in order to get this thing to be installed because you need core dev. And then other people were saying, well, actually don't use core dev because that requires, you know, composer one and then that's going to cause problems. And then I felt like I went down a little bit of a rabbit hole, because then I'm trying to say, okay, well, actually what I need to do is just install all of the modules from core dev in order to get Upgrade Status module installed and uninstall Drush and then get Drupal Upgrade Status. And I got everything working in all the composer things sorted out, and then I went to run it in like the drush commands that came with it didn't work.

Matthew Tift: And then there were more issues. So I mean, again, I it other sites, this might just work flawlessly and I might have had one particularly bad site, but I just want to sort of warn folks that something as easy as installing a module to tell you what to upgrade could be fraught with some challenges. And there are ways around it, and a lot of smart people are figuring this all out. But these are the kinds of things you will hear people talking about. If you start asking. They might say, well, I'll just use Drupal check, or just use Drupal Rector or Upgrade Status, or don't use core dev. And and there are various opinions on these things. And one other tricky aspect of that is people have various opinions on these things, and those tools keep changing to keep up with their dependencies on other projects. So the solutions keep changing. And. And in some cases sort of quickly become out of date. So again everybody's going to have a different experience depending on what they have installed on their site and their requirements and all of that. But there are some there are some things that might look kind of funky, like install, uninstall drush in order to install this other thing in order to install this other thing is not something I would normally say makes a lot of sense, but that is one of the like particular things that I've encountered trying to upgrade to Drupal nine that seems to deviate from the oh, it'll be really easy to upgrade from Drupal eight to Drupal nine point of view.

David Burns: We've experienced a bit of that in support. Right? So we're working with a lot of different code bases, and they all have some of these dependency differences between them. But we're starting to see a pattern and we're starting to like really narrow down what it takes to get to a point that those Upgrade Status and Upgrade Rector will work. I think that's that's like one of the benefits is if it's a one off time and you're kind of learning it from the get go, it's probably going to take a fair amount of time if you're running into these things. But if you work with somebody that has done 3 or 4 upgrades you know, there's there's a lot to be shared with, like the knowledge gained along the way.

Matt Kleve: I'm actually surprised to hear your experience, Matthew, because it wasn't mine at all. So it's it's good to know that your mileage may vary, I guess. And again, I warn if you haven't looked at this yet, you should probably look at it now. The site I was on already had Upgrade Status installed. When I was told, hey, Matt, go figure this out. So I was able to add the, you know, what was it, the Palantir Rector Library and the Upgrade Rector module and really start jumping into things and solving problems. So sounds like every site's a little different and that's something to be aware of, I suppose.

Matthew Tift: Yeah, I would, I would add additionally that this particular site I was working on had taken the approach that some people have used in the past. That could have been a legitimate approach in particular situations, which was to only upgrade their modules for security releases. And that maybe was a legitimate approach in the past. But some of the issues that I was encountering were because this particular site had a lot of modules that were far out of date. They didn't have any security releases, but the modules themselves were out of date, which was creating those composer issues because the the newer versions of some things had required newer had dependencies that were newer, and that was where the things were coming from. And ultimately, I think with that particular site, the solution was first, upgrade all of your contrib modules. And I think that is an important thing that I could say definitively, not just like, oh, you might this, but I would say make sure all of your contrib modules are up to date in terms of the ones at least you know you need before getting ready to do maybe even that, that that next step, that step of upgrading or installing Upgrade Status.

Matt Kleve: And that's actually where I ran into my most problems with the Drupal nine upgrade is the eight site that I was working with was built early in the eight cycle, so a lot of the contrib that was being used was a version that isn't necessarily supported or desirable at this point. So when you're using old code and then custom code was built on top of the old code, upgrading the module screws up your custom code and then your head explodes trying to figure out what's going on.

David Burns: So I've got a great real world example of this. One of the sites we did upgrade, I looked at it initially and it was like ten contrib modules and four custom modules, and I'm just like, it's going to take like less than a week, right? But when we started getting into it, we realized that, number one, the core was still running a very, very old version of eight, I believe, 8.4. And we needed to get that up to 8.9. And then because of that, we had to do like the switch from media entity to core media. And then we had a handful of modules which were not widely used, like Exif module and media image Exif and like another one called Google Vision. And there's very little usage of those modules and no patches that made those available to D9. So the smallest, easiest site I thought actually took the longest. And we have sites that have hundreds of contrib modules and 50 custom modules. And they were much easier than this super small site.

Matt Kleve: Wow.

Cathy Theys: I think a lot of this is, is, is like it might be easy, it might be hard, and there's no way to tell until you dig into it. So dig in now.

Matt Kleve: Dig in a month ago. Dig in six months ago. But...

David Burns: Yeah.

Mike Herchel: So I have a question regarding composer. So when Drupal eight came out like some people were installing it without composer, some people weren't selling it with composer. And then there was like the Drupal or there was the Drupal Composer Scaffold thing that was on GitHub. And then eventually at some point, I think it was like 8.7 or 8.8, they did Drupal composer Scaffold went into core. And how did how how difficult is it to kind of upgrade from that Drupal, Drupal composer Scaffold thing to Drupal nine? Because I know Drupal Composer Scaffold is kind of deprecated.

David Burns: So Scaffold is still used. And what Scaffold does is or maybe we're still using it, but it's baked into core and we're kind of like supporting it in both places. But Scaffold allows you to drop things that are being added into composer into specific folders. That's like not a functionality that ships out of the box with composer that I'm aware of. And I forget what the other part of the question was there, Mike.

Mike Herchel: I'm just looking for, like, sites that were built using the Drupal Composer, Drupal Scaffold. How is it difficult to to migrate those to like the standard core composer?

David Burns: Cathy, you're doing a bit of this right now, and I think there's a story around one specific module that I think you could talk about that shows the benefits of going to a composer workflow. And then maybe like some of the efforts of getting a non composer site to use composer.

Cathy Theys: I might need a better hint than that. I was thinking that Mike's question was like, can you even move a Drupal eight site to Drupal nine if it's not using composer at all? Right? Because one of the things Mike said was like when when Drupal eight first came out, there was like the option to not use composer. And so my assumption is that that, that you can't use Drupal nine without composer. Does anybody know for sure?

David Burns: I definitely wouldn't recommend it.

Mike Herchel: I mean, there's there's a is there a tarball for it? I'm just going to Google it right now.

David Burns: I mean, there's no functional reason why you couldn't, you know, expand modules and drop them into where Drupal is expecting to see them. The one module I was hinting towards was the Address module. Right? Address has...

Cathy Theys: Ah, right because it has a dependency. It assumes you're using composer. So like Drupal core itself might be technically possible to use without composer, but some contrib modules in those contrib module maintainers have decided to rely on composer to manage their contrib dependencies. And so that can be quite tricky. Yeah. And the Address module is a good example of that.

David Burns: And not just contrib dependencies like there's external libraries that don't exist in the Drupal ecosystem, right? So through package management through composer it's able to go out, grab those and drop them into, you know, the places that the module is expecting to see those dependencies like the libraries folder. And if there's even instructions on the address module, that's like if you're not using composer, these are the steps you have to do. You have to actually go find those libraries, unzip them, drop them in. And just using composer takes a lot of that chance of like, human error out or just doing something wrong and deploying it and something not working.

Mike Herchel: So there is there are still like tarballs and zip files of Drupal nine on Drupal.org.

David Burns: And that's actually what composer is pulling in, is the tar files and unzipping them into the vendor directory and then copying them to where they belong within Drupal.

Mike Herchel: Yeah, but composer doesn't. Composer isn't pulling down. Correct me if I'm wrong. Composer isn't just pulling down the Drupal 9.2.4 tarball. It's actually pulling in like the the the Drupal 9.2.4 tag from git looking at that composer info, reaching out? No?

David Burns: No, it actually pulls the compiled version. So if you open the composer lock file...

Mike Herchel: Okay.

David Burns: There's a lot of info in there. And one of the things is, this is the source that composer pulled the package down from. And the reason we know that is if we open the info.yml file, we will see additional content in that file that's not seen if you go to git drupal.org. There's a process that when you cut a release on Drupal.org, it actually compiles that tarball and inserts additional information about like a timestamp of when that was created and the version number for that. I've went down a long, long rabbit hole of understanding how composer works.

Mike Herchel: I'm happy to be a front end developer. [Laughter]

Matt Kleve: Composer has saved a sum total of zero minutes for developers ever. Don't at me.

Cathy Theys: Oo hot take!

Mike Herchel: Boo! [Laughter]

David Burns: I will say. I will say, ever since Drupal eight began and I've been in the support and maintenance department. I would say half of my time is spent dealing with composer, whether it's just updating packages or trying to figure out some of these conflicts. And less time in actual, like module and bug fixes.

Mike Herchel: That's interesting.

Matt Kleve: And that's where we're going to get into right after this. Coming up, we're going to talk about the things that Rector actually will solve for you, code changes you may need to be looking out for, as well as some interesting changes that you might find on the front end of your Drupal nine site coming up right after this.

[Intro music].

Matt Kleve: And just one second, when we step aside from the podcast, we want to tell you about something that Lullabot is doing. And it has to do with this very topic, doesn't it? Mike Hershel.

Mike Herchel: Yep. Upgrading Drupal.

Matt Kleve: And I hear you're one of the experts that's going to be speaking, right?

Mike Herchel: Well, expert is kind of a relative term, Matt. [Laughter]

Matt Kleve: You will be joining Cathy Theys, who's also been on this podcast, you'll be joined by another front end developer at Lullabot, Andy Bloom. Lullabot will be hosting a webinar. There'll be a live Q&A session. Maybe you have some questions that we didn't get to on the podcast, and you want to get them answered. You'll have the chance to do it right on the webinar, right?

Mike Herchel: Yep. So like any type of follow up questions, any type of upgrading issues any anything from this podcast, just feel free to, to hit us up on on that webinar.

Matt Kleve: It's like we planned this whole marketing strategy or something, didn't we?

Mike Herchel: Well, maybe not you and I, but somebody.

Matt Kleve: [Laughter] That's right. So if you want to register for the webinar, it's free. If you go to Lullabot's Twitter, that's the easiest way I found to do it. If you go to twitter.com/lullabot, one of the pinned tweets at the top is for the webinar registration. It takes you to a Zoom website where you put in your name and your email address and stuff, and you'll get the the link to the webinar that's happening September 23rd at 1:00 in the afternoon eastern time.

David Burns: Yep.

Mike Herchel: And I'm looking forward to it.

Matt Kleve: And we want you to be there, too.

Mike Herchel: Yes we do.

[Intro music]

Mike Herchel: Welcome back. We're talking on the Lullabot podcast about Drupal eight to Drupal nine upgrade and how it should be easy, might be easy, but sometimes can be slightly, slightly difficult. So, Dave, you mentioned that it's very important to keep your module modules upgraded. Is that the first step? And this question isn't necessarily directed at you, but.

David Burns: I will say the sites that have been doing a good job of maintaining their updates is much easier.

Cathy Theys: Yeah, it totally is. It's and I think that reminds me of something that Karen said the other day, which is like the the promise of the easy upgrade from Drupal eight to Drupal nine. Being as easy as a point release isn't so much like true or false. Like, yeah, it's a lot easier than before, and it kind of could be that easy with the big asterisks caveat that says if you have been keeping your Drupal eight site up to date. Because many projects are going to find that the first step to getting to Drupal nine is updating Drupal eight. And so if if that's been postponed you know, these last couple years, that work still needs to be done. It cannot be skipped. And that is different because with like, let's say going from Drupal seven to Drupal eight, the the way that would be done is to like build a new Drupal eight site, make it have all the features that is needed, and then do a data migration from the previous website into Drupal eight. And so there was nothing really you could do to Drupal seven to make it be Drupal eight. That's like not possible. But in this case, because we're moving from Drupal eight to Drupal nine and the community, you know, made this decision about backwards compatibility and how we're going to move forward, you know, into these next ten years with doing updates. It's totally different now. And so the work can be done inside your Drupal eight site. You don't need to make a Drupal nine site and then migrate all of your stuff and recreate your site in Drupal nine, but you can't skip the part where you keep your Drupal eight site up to date. That work still needs to be done. And yeah. So so that's like, you know, also something that needs to be started as soon as possible.

Matthew Tift: I would say that in a way this is like, say, painting your house where you have to do a lot of prep in certain instances. In other instances, you might have a very nice house that does, you know, that maybe is is less out of date and doesn't need the paint job as, as badly. But it's this issue of like windows scraping and taping and all these other things that you kind of have to do to make your job easier for that last step. So this this might be an imperfect analogy, but really you do you have to do a lot of prep work in Drupal eight. And I think we're talking about like at least three different classes of, of problems that you have to solve. One is your Drupal core update; so if you're mostly using Drupal Core and maybe you don't even use Drush or something like that, this process could be super simple because that process is there's a very clear upgrade path. If you have custom modules that that process in a sense might be the next easiest because there's some good tools like Drupal Check and Drupal Rector that will tell you where you have made your, your errors in terms of your code quality. So you can just make those changes and keep going until the tests pass. And then there's that third, at least that third aspect of this, which is the contributed modules. And the contributed modules that requires you to work with other people in some case, in some cases, be dependent on other people or to create your own fixes, so that in a sense can be something that can hold you back. So getting all those contrib modules is a pretty important, important piece. And there are some good tools I think that we can use to help get there.

David Burns: Yeah. Matthew, you mentioned earlier that a lot some projects only want to do security updates, right? And we've seen projects like that land in support. And we also have other projects that we said, hey, Automatic Updaters Composer allows us to do that. And what these automatic updaters do is basically look at your composer file, look at drupal.org, and every time there's a new release creates a PR for that. And if you're using something like tugboat, you even get like an environment where you can click around and see what that update does, if anything changes. Even better, if you have automated tests to do those, those reviews for you. But the idea here is every time there's a minor point release, you get this module introduced to your code base. And we have two sites that are built on the same platform, the same one that had the 100 custom modules and 150 contrib module or custom modules, 50 and 100 contrib. Sorry, one of those.

Cathy Theys: A lot.

David Burns: A lot of modules. And one of those projects we had been following along and doing all the updates for, and the other one we haven't. So at the end of the day, when we went to each of those sites and said, we're ready to do the Drupal nine update, one of them really only needed like seven modules updated, and three of those contrib modules didn't have a D9 release or a patch on Drupal.org to make it D9 compatible. The other site required about 60 of those contrib modules to be updated. And all those modules, none of them had a security release, right? Because we would have updated by that point. But those minor point releases that were introduced brought those modules up to be D9 compatible. And the three automated updaters I'm going to mention are: dependabot, this is integrated with GitHub, GitHub bot Dependabot, and it's now part of their ecosystem; Violinist.io; and then there's another one called Renovate Bot. Violinist is a service that you sign up for and it you add something to your repo and it's able to watch it. Once you authenticate, renovate bot is like a self-hosted type of service that does the same thing.

Matt Kleve: Interesting. I've only heard of Dependabot because I've been on projects that you check stuff in, and Dependabot yells at you about outdated libraries and you're like, I don't want to touch that library. I'll do that later.

Mike Herchel: Yes, lots of emails,

Cathy Theys: I think...

Matthew Tift: Recall Dave, didn't you do a whole podcast on that subject? Or.

David Burns: My last...

Cathy Theys: I think we...

David Burns: Drupal seven extended support. I don't think I did a podcast. I did the internal show and tell.

Matthew Tift: Oh, okay.

David Burns: Yeah.

Cathy Theys: Future podcast coming.

Matt Kleve: There you go.

David Burns: Stay tuned.

Cathy Theys: Yeah, I think I think this is like getting back to what we were talking about earlier where like, we had a pro, you know, there was a promise in the message of, of the big change that the Drupal project went through when it decided not to do things like Drupal seven and to do things like Drupal eight. Right, like using composer and having dependencies and, you know, hoping that things will get easier. And I think that promise has also itself a lot of dependencies. Like we spend a lot of time fussing with composer, but we could automate that with like Dependabot. We might need to spend time testing those updates, but that could be automated with automatic testing. And so I think the projects that are like seeing that promise of a brighter future realized are the projects who have gone all in on the modernness. They're like, okay, we got it, composer, automated test, automated updaters. And those projects might be seeing some acceleration in the amount of features that they can deliver in a year, even though the developers might still be spending a great deal of time keeping that automation running, like all the DevOps integration, to keep that going is also a significant amount of work.

Matt Kleve: So I think we've covered that all of these build tools can be built differently and cause different problems. For that 8 to 9 upgrade, let's talk about the actual code changes. So once we run something like the Upgrade Rector module, it's going to yell at you and say, hey, there's this list of changes. Do you want me to make them? And it's going to make them if you say yes, right? Do we trust that? Did you trust that?

David Burns: I trust it to do the initial batch of work. And then you load the site and test it as you normally would, and then look for those errors that pop up and you go to Google.

Matt Kleve: Yep.

David Burns: You say, why am I seeing this error? And somebody has probably posted about it either on Drupal.org or StackOverflow or somewhere else.

Matt Kleve: Yes. One thing that you, I should point out is that my experience is there are also some changes that it makes, and it leaves a comment that says something like todo actually confirm that this is what you want to do. Which is kind of useful. But if you're not paying attention and you don't read your diff on this giant change you might not realize happened, right? Beyond that, there are changes that it can't make. A lot of times and one thing that comes to mind, and I remember going through and making a bunch of manual changes was things like the what is it? The entity Manager service in Drupal eight got exploded into 9 or 10 different classes. And a lot of times it isn't necessarily going to know what exactly you want to do?

David Burns: Yeah, it's definitely not going to do a rewrite of an entire code base for you. What it will do is find 1 or 2 lines next to each other that it knows how to do, like a regex search and replace.

Matt Kleve: And a lot of that. Yes. Like what I did find was it did things like the entity manager like the, the most what you wanted to do was you do like an entity manager, get the node storage and then load a node. And it was able to make that change for you. But if you ended up using that entity manager class to do other things like node display that split off into another class, and you're going to have to actually go in there and manually change the code. So...

Matthew Tift: And I definitely recommend breaking things into chunks and not just saying I'm going to do it all, you know, and assume that that works. I mean, that is one approach where you you just kind of...

Matt Kleve: Send it! [Laughter]

Matthew Tift: Maybe essentially you just run it all and see what's what, see what happens. You know, maybe it just works. And I mean, that is an approach. But I would definitely say if you want to understand your code base a little bit better break it into, you know, functionality that you could then do some manual testing on or something along.

David Burns: And that's. That's the benefit of the reverse compatibility, right? Because you can only change one module, get it prepared for D9 and not have it be nine. And that code should still function the same way it currently has with the latest version of Drupal 8.9.

Matt Kleve: A lot of my coworkers might laugh because I've been fairly outspoken and critical about a lot of automated testing. Depending on how it's used within your code base. But automated testing is definitely useful when you're making changes on this scale, because it's it's that initial smoke test. You change all these things and they should be minor changes, but you change a lot of them. Do your tests still pass? Yes. Okay. Chances are you didn't break a whole lot. So that's a good thing to know. I think having tests is great and definitely helpful in the process.

Matthew Tift: Yeah. Part of what you're doing. I mean, I know we've said this in different ways, but I don't know how clear we've made it. But you're let's say you're updating a custom module. You're making that module compatible with both Drupal eight and Drupal nine at the same time. So that's why you can do a lot of the testing already on Drupal eight. And then when you have all of those working, all of those tests passing on Drupal eight, that's when you do the super easy. Oh, it is just like a point release because you've done all that prep work, like the prep work for painting a house. Now, I guess when you paint a house, there's actually a lot of work to painting the house. So my, my my simile is falling apart there, but you get the idea that there's this, there's this sense of you're doing this prep work on the same, you know, code base. Now, that's if you're going from Drupal 8 to 9, if you're a 7 to 9 is a whole other can of worms. And that's a lot like 7 to 8. But you also have the added thing of making sure those modules are compatible with Drupal nine when you get there. And and for the most part, a lot. You know, all the big modules, I think I shouldn't say all, but a lot of the big modules are all Drupal nine compatible. But there are still a few where, you know, I've run into some weird issues, like I wanted to use this, this date calendar thing, and it didn't quite work the same in Drupal nine, so it was just easier to upgrade the site to Drupal eight, except for that one thing.

Matthew Tift: And then wait, you know, that was my approach. And then wait. That's the one thing. I still can't get it to work, but you can sort of bump off like you can do. You can do 95% of the work and then, oh, this contrib module isn't quite there. You you could probably not feel like you have to do that today. You know, there are probably a lot of other people that, you know, are going to try and get this fixed within the next three months. And some people that that might be more important for them than for you. So you can sort of pick and choose, you can pick away at it. But, you know, I think what when when Matt was saying before, like, get this ready yesterday or done, it's more like do the testing now to get the scope of the project. And it might take a while just to get to getting the scope, but it might be that, you know, like we like we've been saying it could be it could be that somebody else was already sort of preparing for this for a long time. And your site's basically ready.

David Burns: There's also been a robot that's going through Drupal.org issue queues, and from popular modules that haven't been upgraded to deny it'll run Drupal Rector, take that patch and automatically create an issue that even if it doesn't get all the things, a developer could go get that, import it, and then figure out what the remaining pieces are, and then add to that issue to get the module even closer to D9 and the community can review it and all the maintainer has to do is merge it in, cut a release and they will have a D9 compatible module.

Matt Kleve: So the actual work of painting the house that changing the code I would argue, is a lot like a Drupal 6 to 7 upgrade. There were some API changes. The bulk of it was similar. But you do kind of have to be aware about the of the new changes in the versions. So 8 to 9 in my opinion, is a lot like 6 to 7. The basic structure is still the same. But we're going to do a few things a little different way.

David Burns: I'd agree with that.

Cathy Theys: Yeah.

David Burns: We've been talking about preparing your code base and all the contrib and custom modules. One thing I've found out is you can't actually do that with the theme, because Drupal nine requires Twig 2. And if you're still running Drupal eight, Twig 2 is not available to process the theme, and maybe there's a solution around that...

Mike Herchel: That's not exactly true.

David Burns: There we go.

Mike Herchel: Yeah. So Drupal eight supports Twig 2.

Matt Kleve: I knew we kept you around for a reason, Mike.

Mike Herchel: Yeah, yeah, yeah. Drupal eight supports Twig 2, so you might already be running Twig 2 on Drupal eight and not even know it, depending on what like composer has done. And someone in in the comments correct me if I'm wrong, but I'm pretty confident it does. In my opinion, like running upgrading a theme from Drupal eight to Drupal nine is really easy. I haven't had to do anything. I think at one point I had to google something, you know, threw an error and I googled it, and then it was just like a minor Twig change. I know there's a couple of things that are deprecated. There's like the raw filter, which hopefully you're not using anyway because that could that could lead to security issues. That's been replaced with verbatim. There is a a document, a a one page documentation on Drupal.org that we'll link to in the show notes. And it just has a couple minor changes to blocks and things like that. You can also look at changes on within Twig's documentation of upgrading from Twig 1 to Twig 2. But honestly, there's not a heck of a lot going on. There's not that much to change.

David Burns: Does this include the package management and some of the tooling around compiling your theme, or is that specific to using design libraries like Pattern Lab or Fractal?

Mike Herchel: Yeah. So if you're using something like Pattern Lab, especially Pattern Lab version two I think only works with twig version one, and that can open up a big can of worms pattern like the state of pattern libraries for for for PHP Twig is is not, in my opinion, great right now. I know there's there's a number of systems that use twig JS, which isn't it's very close to twig PHP twig, but it's not quite the same. So a lot of people try have tried to use Pattern Lab 2. But yeah, that can be an issue. But if you're not using like any type of pattern lab 2 or anything like that, you know, you mentioned like things as far as, like compiling your, your, your sass or JavaScript or something like that. That's not dependent on Drupal. That's dependent on the theme. So you don't you don't need to upgrade that.

David Burns: Cool.

Matt Kleve: The one change I found and Mike rolls his eyes every time I tell him this, but I was proud that I had to find this. Is that same as indivisible by from twig 1 to 2. Those two tests are two words instead of one word. And that was.

Mike Herchel: There you go!

Matt Kleve: Yeah. That's that's my front end knowledge for you.

Mike Herchel: Good. Good job. I didn't even know that. But yeah, I'm seeing I'm actually seeing like, now that you're mentioning that I'm looking at that drupal.org documentation and I, and I see that and then there's like another, another usage it says as of Twig.1x using the _self global variable is deprecated, so.

David Burns: This is probably worth the whole podcast on its own. But because of the pattern lab changes and stuff, Is that something that you would continue to recommend? And do we foresee that getting any easier, say Drupal nine to Drupal ten?

Mike Herchel: Yeah, that's I don't know if there's. So I know a lot of a lot of front end folks in the Drupal community are using Emulsify my understand, which is the theme. I think it's I think it's primarily developed by Four Kitchens. I'm not too familiar with it, but it integrates in with Storybook. And my understanding, though, is that it uses TwigJS. I have a little bit of experience working with a Pattern Library called Fractal that uses TwigJS, and there's like a little bit of weirdness around TwigJS that, you know, I would occasionally run into some weird, weird bugs that like, especially regarding attributes and stuff like that within TwigJS that, you know, was handled properly in PHPTwig. And then there's there was a couple other things at the time, and I have no clue if this is fixed where it wasn't happening, it wasn't handling the ternary operators exactly like you couldn't do everything that you could do in PHPTwig.

Matt Kleve: I've been bugging you to do a Pattern Library podcast, Mike. Just thought I'd throw that out there.

Mike Herchel: Yeah. I've been trying to

Matt Kleve: @MikeHershel on Twitter. You can bother...

Mike Herchel: [Laughter] I've been trying to. And cc justafish for Sally Young because she says she has, like a halfway working Storybook, php, Twig integration. And so if we could somehow bug her to actually fix that or make that good, I think that would be a big benefit. She's also a maintainer of TwigJS is my understanding too.

Matt Kleve: Neat.

Matthew Tift: You know, I think there is one thing that I don't believe we have mentioned, which is why we're talking about this now. And I think we might have mentioned that Drupal nine comes out in November 2021.

Matt Kleve: Drupal nine is out now.

Matthew Tift: Oh, I'm sorry that.

Matt Kleve: You can be using Drupal nine now.

Matthew Tift: ..that we have to do our up- I put that, I said that very incorrectly, that we have to do our upgrade from Drupal eight to Drupal nine in November 2021.

Matt Kleve: You definitely want to. Yes, yes.

Matthew Tift: And the reason for that isn't just like some of these dates in the past in the Drupal community where we've said, oh, we'll wait till it's done, we'll wait till it's done. We actually have a hard deadline. And that's because Drupal nine uses Symphony 4.4 and Drupal eight uses Symphony 3. I think those numbers are the correct ones. So the the Symphony 3's end of life is November 2021. So when we've said all these things about like composer dependencies and other dependencies, well, this that's the reason why we have this hard deadline. It's not a soft deadline. It's it's a real deadline. So in addition to the other things where it's like Drupal nine uses Twig 2, Drupal eight uses Twig 1, or, you know, Drupal nine using PHP unit eight. There's there's a lot of other things, but it's really I would say it comes down to one of the key reasons is it comes down to this end of life for Symphony 3 in November 2021. So that's why we have this hard deadline. So I think that's another point worth emphasizing that it is a hard deadline. And that's that is the deadline. And and that's why. So I'm you know, there are I want to throw in a million caveats, but I won't for now. But that's the deadline.

Matt Kleve: So we fought for a long time to get off the Drupal Island. And now we have to play nice with our neighbors. Right?

David Burns: And I just want to say, like, your website's still going to work when Drupal is end of life.

Matt Kleve: Yeah! It's not it's not going to quit.

David Burns: Symphony 3 isn't going away, and your composer builds aren't suddenly going to fail. I don't think I'm pretty sure they won't. But the recommendation there is Symphony 3 will not be getting security updates. So if there's an exploit in PHP or the Symphony library framework, nobody's going to be looking at that and writing any fixes for it. So do your best to get your site up to Drupal nine by November, or within a few months after that.

Matt Kleve: You mean get it done.

David Burns: Get it done.

Mike Herchel: Get it done.

[Laughter]

Cathy Theys: And and at least take a look and, you know, give it a try so that you might be able to see. Will it take a lot of time or a little bit of time? Like that's, that's really like your first step, right? First step.

Matthew Tift: Or I mean, or if you're the kind of person who's still running on Drupal 4.7 and you're like, I can figure this out and I can do my own security updates. Absolutely. You know, you can keep doing fixing things on your site and, you know, not use Composer in Drupal eight. I mean, this is when I was saying there's a million caveats that you could say to what I said before, but for, for for the ease of use or if you're doing this professionally, which our studies tell us and our data tells us a lot of a lot of us are doing this professionally now. Then the professional thing to do in in most cases, again, there's there's reasons why you wouldn't do this. And people have, you know, they can't commit time and money and whatnot at now but they could later. So there's, you know, there's all kinds of there's all kinds of things. But I just I think it's still worth noting that, yes, it will still work after then, but that's that's why we have this deadline. That's why we're talking about this now comes back to the Symphony end of life Symphony 3.

David Burns: And if you're finding it challenging to get to Drupal nine, still focus on getting that done first. But then the conversation right after that is, how can we make this easier next time? And that goes back to the conversation of automated updates and automated testing. To keep your site as up to date in preparation for when Drupal ten comes around.

Matt Kleve: We kind of Pooh poohed the idea that this was not as easy as we were told it was going to be. Or at least we spouted the idea that this was harder than we were promised when we talked about this originally, or at least it was discussed within the community. But I guess I kind of want to point out that any 7 to 8 upgrade that I've been a part of has been a complete migration and a rewrite of all the custom code. It is still easier than that. Like by far.

David Burns: Significantly.

Matt Kleve: So I guess if that's what you're comparing it to, then it's a walk in the park. But...

David Burns: And the contrib ecosystem came along with it much faster. Like to get popular modules D8 compatible took some of them two years after D7 was released. And I'm not seeing that happen with Drupal nine.

Matt Kleve: And I think it's because within one module, getting one module up to date with Drupal nine was a whole lot easier. Like, it wasn't the complete rewrite. It wasn't...

David Burns: Just deprecation.

Matt Kleve: ... Let's figure out all of this object oriented patterns that have been adopted and I wasn't paying attention.

Matthew Tift: Yeah.

Matt Kleve: This was a lot smaller of a change, but still a significant change from what I've seen.

Matthew Tift: Yeah, the the Drupal seven to Drupal eight was ripping out the guts of Drupal and replacing it with Symphony. So that's that's what we talk about when we're talking about the dependencies. And it it if you have been the kind of developer, if you're a developer and you're writing back end code and you ignore those little warnings about this will be deprecated, or it's, you know what, I just want to keep using my hooks, or I just want to keep using this function or whatever it might be that still works in Drupal eight. Then then it's going to be a little bit more work. If you've been up to date on not using those and you've been keeping your modules up to date, and it could, you know, like we've said, it could just be a walk in the park, but you know, on the other hand, if you really know Drupal well and you have a specific skill set, I want to throw this out there and say, like, you know, my imposter syndrome ratcheted up a few notches when I thought, oh, I can't even install a module.

Matthew Tift: So I wanted to at least say, you're not alone if you're running into issues. And I talk to a lot of other developers who who told me like they had to do crazy things like, oh, actually, you know, I had an upgrade script and it had you had to run this command four times, you know, otherwise that one thing that you're updating didn't work or I, you know, I think I was talking with someone in this room where they were like, actually, I had to pin Drupal Upgrade Status to like the 3.0 version to make it work on this site just to, to get to that next step. So people are I've seen people do some really kind of crazy things. And it was it was sort of reassuring for me to just hear others say, yeah, in my case, with this particular set of my own problems in the way that this particular site has been altered, which isn't good or bad. It's just, you know, they made particular choices. It was more it was more tricky, and it took more time than I expected.

Matt Kleve: David, do you have any final thoughts as we point toward wrapping this up?

David Burns: Yeah, I just want to thank everybody that was involved of getting Drupal.org's code base onto GitLab, because I feel that is part of what made getting these modules updated a bit easier for the community. Not just the fact that, you know, we're removing deprecated code, but a lot of the tooling and the decisions around that. So huge thanks.

Matt Kleve: Matthew Tift, any final thoughts?

Matthew Tift: Yeah, I think it's I want to say something similar that the some of these projects that we've talked about encountering or I've talked about encountering challenges like the Upgrade Status module. I am extremely thankful for Gabor and and WebChick and some of these other folks that that have worked so far. I can't name all of them, but there have been a lot of people that have worked to make this process as easy as possible, and I am I am incredibly grateful, and I just want to say thank you to them for these tools. The issue, there's no way that people making these tools can predict all of the different ways that we hack away on Drupal and the changes that we make. So I want to express some deep gratitude for those folks that have made it a very easy process for quite a few people. And that's that's where I'd like to stop.

Matt Kleve: Cathy, any final thoughts?

Cathy Theys: Yeah. I think it, for people who are running into issues there's drupal.org issue queue is good. Slack is also good. If you're not in the Drupal Slack yet there's a page drupal.org/slack, which has the instructions for joining the slack. And then once once you're in there there's the support channel and a D9 readiness channel also in there. And you might start out in support and then get redirected to another channel, because maybe your question is about composer specifically, or maybe it's about Upgrade Status, or maybe it's about, you know, a module update. So you might get redirected. But those are good starting points.

Matt Kleve: I always forget about Drupal Slack support. It's been a while. That's good stuff there. Mike Herchel update your darn site, huh?

Mike Herchel: A back end developer on this podcast, so I appreciate you all.

Matt Kleve: You're out of place, but yet in place because turns out we have to care about Twig, too.

Mike Herchel: Yeah, a little bit. I like, in my experience, I did do, like what? Like I, I did a development of a personal site in 8.9 and migrated that to 9.0 at one point, so. So that was my experience and it went fairly smooth.

David Burns: Matt was the Twig 2 a pun? Is it the t-o-o the t-w-o?

Matt Kleve: The world may never know.

[Laughter].

Matt Kleve: Thanks everybody!

Mike Herchel: Thank you.

David Burns: Thanks everybody.

Mike Herchel: Bye everybody.

Cathy Theys: Thanks!

Matthew Tift: Bye. Have a good one.

Cathy Theys: Bye.

[Intro music]