Drupal Automatic Updates—The Update

Matt Kleve: For August 18th, 2022. It's the Lullabot podcast.

[Intro music]

Matt Kleve: Hey everybody, it's the Lullabot Podcast episode 258. I'm Matt Kleve, a senior developer at Lullabot, and we're here to talk about Drupal. Lullabot is a digital agency: strategy, design, development. We work primarily in the Drupal world, a Content Management System framework that builds websites and does great things. So today is kind of a follow up podcast, something that we had on a while back talking about automatic updates in Drupal. And with us today, we've got a few folks that have been involved at making that happen and, you know, getting that closer to a reality. First up, we have someone who's been nearly 14.5 years on Drupal.org. He's been credited with many fixed issues in Drupal core and contrib. I'm told he that he is the most prominent Drupal developer born in Gainesville, Florida. Somebody might disagree, but I'm not sure. He's a software engineer at Acquia's Drupal Acceleration team from Ithaca, New York. We have Ted Bowman. Hi, Ted.

Ted Bowman: Hi. Thanks for having me. Yeah it's good to be on.

Matt Kleve: Welcome back to the podcast.

Ted Bowman: Yeah.

Matt Kleve: We also have someone else who's returning to the podcast. He's been on the Lullabot podcast a few times, episode 207, 222, 243. Known as Hestenet on Drupal.org. He's the CTO of the Drupal Association from Portland, Oregon. We have Tim Lehnen.

Tim Lehnen: Hey, thanks for hav-

Matt Kleve: Did I get, Lehnen? Lehnen? Did I say that right? Awesome.

Tim Lehnen: Yep. You did.

Matt Kleve: I was thinking I was, I wasn't. Yeah.

Tim Lehnen: [Laughing] No, you nailed it. You nailed it. Glad to be here. Glad to be a part of the podcast again.

Matt Kleve: And and a newcomer to the Lullabot podcast. We have someone who's been on Drupal.org for nearly 14 years. He's the co-maintainer of the Aegir hosting system and contrib components. It's a web hosting control panel program that provides a graphical interface designed to simplify deploying and managing Drupal, WordPress, CiviCRM websites. He's a founding partner at Consensus Enterprises from Montreal, Montreal, Quebec, Canada. We have Christopher Gervais. Hi, Christopher.

Christopher Gervais: Hi, Matt. Nice to be here.

Matt Kleve: Glad you're here. So, where are we since we left off? I remember we had a podcast, gosh, it's been a, I don't know, over a year or so ago. Right? And there was work done toward adding automatic updates to the Drupal-sphere. And I remember there were some ideas and kind of some groundwork, and now we're got a bunch of work done and something is kind of functional, right?

Ted Bowman: Yeah. We have a contrib module stable version out. So it updates Drupal Core as sort of the Non-experimental stable feature. Right now it's just through a form, like it's not a while you sleep kind of thing, but you go to a form you're shown you know, what you can update to, you choose it, and hopefully it updates. And there's not like a lot of configuration. That's sort of what we're going for.

Matt Kleve: So it's not exactly automatic yet. It sounds like there's still manual intervention, but it's it's better than it was? Right?

Ted Bowman: Yeah. I mean, I guess I think of it as you have an automatic dishwasher, but you still have to press the button to wash your dishes.

Matt Kleve: That's true [laughing]. I, I would like the world where I didn't have to, like, load the dishwasher and push the button. That would be awesome.

Ted Bowman: Yeah. Well, in this case, you don't have to load the the in that analogy, you don't have to load the dishwasher. It's just like the dishes are magically put into your dishwasher, but you do have to press the button.

Matt Kleve: So back when we talked about this on the two podcast number 243 we kind of rehashed some of these ideas, but I think it's important we bring them up again. Why is this something that's important to the to Drupal that we have automatic updates?

Tim Lehnen: I mean, I think that's a really good question. And there's there's multiple reasons, right? So we are probably going to cover some of the same ground. But one of the primary reasons is that the sort of ease of use of Drupal and ease of choice in choosing to adopt Drupal, right? The capability of having your site automatically update itself is going to save people a lot of time, hopefully give people peace of mind when they're thinking about how to, you know, maintain and manage their websites. There's a security component as well, right? If patch releases can be fully automated and you can get your security updates while you sleep, you know, that's going to be better not just for the site owners and the teams maintaining those sites and everybody who hangs out in Drupal Slack on Wednesdays waiting to see if a patch drops. But also for end users out in the world. Drupal is, you know, a dominant platform out on the web. And there's a lot of user data stored in important websites, and keeping all those things more secure more easily is going to be great for everyone.

Matt Kleve: Yeah, I was I was thinking about that that hold your breath on a Wednesday and see see what you need to update. Right? Because I mean, due to the nature of open source, if there's a if there's a bug that needs fixed, everybody's informed about it at the same time, including not necessarily good actors. Right? So.

Tim Lehnen: Yeah. Yeah.

Matt Kleve: Keeping your site up to date is important.

Tim Lehnen: Yeah.

Ted Bowman: Yeah. And we've done a lot of work for the while you sleep updates. It's basically we have a flag turning it off right now. In the contrib module because, you know, and Tim can talk, Tim and Christopher can talk about this, basically we want a sort of a higher level security when nobody's going to be around because potentially, you know, an automatic update system is a new place that people will try to attack. So we are implementing that to be the whole system to be more secure. But we've already, you know, done a lot of the programming on the client side and on the server side for for the extra security that we want. It's just the I guess it was determined by the security team that we want all of that to be done before we actually turn it on for people. So that updates happen in the middle of the night.

Matt Kleve: So a lot of people use Drupal in a lot of different ways. What type of user were you working toward when you started out developing automatic updates in a contrib module?

Ted Bowman: So, yeah, I mean, I guess it's sort of predates me. But I think a lot of the sort of the target audience is, at least for people who are going to use it in production, is, I think, what's often referred to as sort of long tail sites. So by nature of, you know, it updating your code on production, a lot of hosting or a lot of people choose to not allow that to happen. And a lot of people with enterprise sites or sort of larger sites that are mission critical and their, you know, their whole business is developed around the site. They actually don't want an update to happen without them actually looking at it. So for people who are going to use it in production, it's more just sort of the long tail site for people who are okay with it being written in production without them actually seeing it. But if for people who are using it, maybe not in production, but using it as a way to sort of like, well, composer's a pain for me, and it's sort of difficult. That could be anybody, because you might not run that part of it directly on your production server. It may be just in that case, it may be a way for anybody to that finds composer difficult to sort of have a UI to do the things that they do now via composer. But I guess for the target audience it's sort of the people who find it also difficult who don't update their sites on the Wednesday, right? Because, you know, you talked about people waiting up on Wednesday to update their sites. A lot of people don't do that now, and they just update it when they when they get to it. And we would like it easier for them to you know, one, it's like easier for people outside of the, the time zones that we the security window time zone on the other side of the world, it'll be easier for them. But in general, people who for whatever reason, find it difficult to keep up with security updates that they won't have to think about it that much.

Matt Kleve: Yeah, it's been a while since I've thought about this, but yeah, those security releases are eastern time business hours or something like that.

Christopher Gervais: Roughly. Yeah.

Ted Bowman: Yeah. Yeah.

Matt Kleve: So kind of in that world is what they try and do. And if if you're running sites in Singapore or something, that might not be great for you.

Tim Lehnen: And I think. Another, another sort of insight about the audience. Right. This goes back to a little bit about what we said about why, you know, why does Drupal need automatic updates, among other things, in like multiple product surveys, right? It's been one of the most asked for features because there is a whole category of users who and this might be sort of small business users, nonprofits this might be people kind of in terms of scale, there might be on the lower end, but in terms of ambition, of what they want to do with their site, they may be on the higher end. But they don't have the resources to have a team to, to kind of manage their update workflow and all of that kind of stuff. And they, they, they want something that lets them get closer to being able to use Drupal as a kind of set and forget project. I think what's also interesting when talking about the audience is if we look way back into the kind of history of this project the very first real movement in terms of, you know, lines of code written in the direction of, of automatic updates for Drupal the sort of 1.0 version of this contrib module was actually written back for for Drupal Seven use cases because the European Commission reached out to the Drupal Association and said, hey, we want those sorts of capabilities to exist because we have a big suite of of sites that are all on this platform that we need to support. And, you know, whether they ultimately actually run those in production? Probably not. But, you know, as as Ted said, they can run those things in their kind of dev and staging environments, run additional CI on that, and then be ready to know that they can push that to prod really quickly without having to, you know, wait for the release and then test to make sure it's all going to work. So, I don't know there's a lot of folks and I think as the capabilities get stronger as we move beyond just updating core to adding contrib capability to adding I don't know, major version capability eventually, who knows? It'll be even more widely applicable.

Matt Kleve: So how does this work? I'm used to these, you know, giant websites that are because that's what I've been working on now, you know, at Lullabot for ten plus years, is these giant websites that are now, you know, very tied to composer and that kind of thing. How does the contrib module work with a site like that? Will it update Drupal for me?

Ted Bowman: Yeah. So it is composer aware. So the idea is we it will run a composer command. It actually stages because of the target audience, we sort of want it to work if you only have one environment. So it'll it'll copy your code base, not all of your, like, uploaded files or whatever, but everything that's managed by composer into another directory, runs the composer command there, whether it's like right now, what we support in the stable version is core, and we have an experimental submodule for contrib, but it generally just runs whatever composer command you send it in the staging environment. Then we can check certain things about it. So one example is for updating core. Right now the core update just supports core updates. So if composer happens to update a contrib module we're going to say, okay, we don't support that right now. You need to take care of that update by itself. And so there's some things we can check. We can't actually fully bootstrap Drupal, but we know what the code base looks like, and we know how to check certain things. And then once everything is okay, the user if, you know, if they're doing it through the form, they would get a form telling them, maybe, okay, you're going to have to run database updates after this because we detected a database update and then they can hit confirm.

Ted Bowman: Or maybe it's not a good time to run database updates so they can be like, okay, I'm going to wait until an off hours and then run this update. So yeah, basically after after it runs the staged update in the runs the composer command in the staged area, then it would it would sort of copy everything back over and your site would be minimally you know, not we'd put it in the maintenance mode minimally, just for a really short period of time, not for the whole composer operation and the checking that everything's okay. So yeah, it is composer aware. Your composer JSON or log file is going to be updated. And afterwards the code base would look like you would run the composer command directly on it. It's nothing. You know, there'd be nothing, particularly after the fact that would tell you. Oh, this was an automatic update composer update versus just a regular composer update.

Matt Kleve: Okay, so it sounds like it's it's doing a lot of the things that a person would end up doing to update the site anyway.

Ted Bowman: Yeah, yeah.

Matt Kleve: It's just that the site is now self-aware and knows it needs updated.

Ted Bowman: Yeah.

Matt Kleve: So it goes and gets it done.

Tim Lehnen: Yeah, and there was a lot of work that Ted and the team on that side. There's, you know, a large number of contributors who participated in this. There's a lot of work they did to kind of figure out the order of operations of doing this and how to do it in a number of requests, right? Half the point of doing a separate staging environment for the composer update is right, if there's some network failure or some other issue midway through, it's not going to leave the site in a broken state... to make sure...

Matt Kleve: Explain if you would real quick what that what that means. I heard Ted say that earlier, and I'm not sure I fully grasped what the separate staging environment is. I heard we were copying the files over. Is that I mean, is it a full environment? Is it just a backup? Like, what are we talking about?

Ted Bowman: So it's not, yeah. It's not a full environment as far as, like a fully bootstrapped Drupal, but it would be everything like in your vendor directory, all of your core, all your contrib projects, basically everything that composer itself manages.

Matt Kleve: I gotcha.

Ted Bowman: We would not copy over all of the the database. We would not make a copy of the uploaded files or your CSS. Except if it's like in the module, but all of, like, the temporary stuff we would not copy over, but then we would run composer on that temporary directory. So composer at that point would be from composer's perspective, it's the same project. Because all the code that it knows about would, would be there. Does that answer your question?

Matt Kleve: Yeah, absolutely. Yes.

Ted Bowman: Yeah.

Matt Kleve: I was just fully kind of understanding. So.

Ted Bowman: Yeah.

Matt Kleve: Gotcha. Cool.

Ted Bowman: So then a lot of things could fail, right? With composer, like Tim said, it could just be a timeout issue or whatever, or it could update stuff that you don't expect. So.

Matt Kleve: Well sure it's connected to lots of things like that's kind of the point of composer. It goes out and gets things and who knows what they might find when they get there. So yeah.

Ted Bowman: Yeah. So we don't want to run that. Sorry. Go ahead, Tim.

Tim Lehnen: I was going to say so. So the idea behind that is, right? Run all of the kind of transient operations, the sort of commands that are getting everything ready, not, even though you are doing this in the live environment, not directly on the live code base. You run them on the copy of the code base so that your last operation doesn't depend on the rest of the internet anymore. It doesn't depend on external dependencies. If everything completed successfully, the last operation that takes your changes lives is just copying those files back over which keeps it as safe as it can be in terms of an update process here.

Ted Bowman: And the contrib like contrib modules, could we just use the Drupal event system can sort of hook into any place, not not using hooks, but actual events can sort of subscribe to any of these events, like right before you apply. It's like I want to check something about the code that the module hadn't thought about or whatever, or there's something particular about your code base that you need to check. So and then they can throw errors and basically say, okay, don't do the update because our custom logic determines that the update is not safe or...

Tim Lehnen: I think that's a really important point because there's a lot of agencies, a lot of like platform hosting providers and things like that, who, you know, maybe they don't want to just use automatic updates functionality off the shelf because they build in their own dev stage workflows and all these other things, but they could actually take leverage a lot of this functionality and then hook into that custom logic, subscribe to those events and maybe integrate it and perhaps use some portion of it to help their customers out as well and maybe save themselves some maintenance time.

Matt Kleve: Yeah, that's what I was trying to like, think about, because usually sites that I'm working on have, you know, the separate environments and your locals. And I was trying to figure out how we could tie something like this into the CI system that already exists. And it sounds like there might be some ways?

Tim Lehnen: I think there's a few options. Yeah, I mean, I can imagine a couple different scenarios and I'm, I'm sure others can as well. But again, you could have a version of your, I don't know if it becomes a whole new environment, you probably don't need a fourth dev, stage, prod, whatever. But you could have...

Matt Kleve: You know, with Tugboat environments are easy. So...

Tim Lehnen: True! That's a fair, fair point actually.

Matt Kleve: [Laughing]

Tim Lehnen: So you could have you could have a branched version of your stage that has updates enabled and then triggers an additional CI pipeline to just validate. And then you come in in the morning and it says, hey, there's an update. And by the way, we already know it passed all of our tests, right? So yeah, ways that we could help out the folks, even even the folks who have that level of expertise and sort of fully managing that pipeline as they do today.

Matt Kleve: So one thing I understand has been a challenge for a long time, and when this was talked about many, many moons ago, what you're required to do is your web server is now going to write a bunch of executable code and run on the internet, right? It sounds like a potentially dangerous situation that you had to get figured out to make safe for everyone, right?

Tim Lehnen: Yeah. There's a there's a potentially. So right. On the one hand, the big advantage of having pervasive automatic updates out in the ecosystem, like we said, is for all those set and forget users, right? This will hopefully help us prevent having sort of zombie sites that are vulnerable to major issues that never got updated. But on the other hand, the updates themselves involve, as you said, overwriting new executable code that's going to get run. So a big problem space is how do you secure that? And a big part of that problem space is how can you have, how can you create trust from a package delivery source like Drupal.org in this case that delivers all of Drupal's code through to a piece of software like Drupal Core or the auto updates contrib module and ensure that that's trusted code, right? Ensure that that's hopefully safe. So this is this is where Chris comes in really because I think there was a lot of work to do. As the Drupal Association, we brought in consensus to help us with understanding how we're going to secure the the kind of package delivery pipeline that Drupal.org uses. And we chose to implement a framework for this called TUF. And I think I'll let Christopher explain exactly what TUF is.

Christopher Gervais: I don't know if I can explain exactly what TUF is in the time we have, but.

Matt Kleve: TUF is TUF is T-U-F, right?

Christopher Gervais: That's right, T-U-F, it stands for The Update Framework.

Matt Kleve: Okay.

Christopher Gervais: It actually comes out of the Python community and sort of academia and finding a Cryptographically secure method of distributing software updates. For whatever it's worth, there's going to be a link in the show notes to sort of an intro blog that I wrote largely just trying to explain some of the fundamentals that that go into it. It's a lot of cryptography basically that tries to avoid problems of like you know, attacker in the middle kind of scenarios or replay attacks, things where if the site, when it's trying to update the request for the updated zip file or whatever is intercepted and a a compromised package is provided that the site has enough context to be able to first identify that that's the case and then stop the update process, alert the user. Things of that nature. And it does so by, well you might be familiar with how like the zip files usually come along with like a SHA256 hash or something like that, right?

Matt Kleve: Yeah.

Christopher Gervais: And generally what that's used for is to make sure that the package you've downloaded hasn't been corrupted during the download.

Matt Kleve: So I can go to a website and download a piece of software or something, and it gives me a hash, and then I can run a command on my computer and hopefully the hashes match. Right? To prove...

Christopher Gervais: Exactly.

Matt Kleve: ... that the software that I am supposed to get is the software I got.

Christopher Gervais: Exactly. And what this does is it provides a secure channel to validate that. Right? So normally you're downloading the the file and finding the hash at the same source. Right? So if somebody were to compromise that site they would be presenting, they could, you know, override the hash with the hash of the compromised package. Right? So what this so what this does is it provides a way where basically we're taking those same kind of hashes, we're making them, we're putting them in a format that in this case, composer, but any TUF enabled client would be able to to use to compare the downloaded file to what it's expecting. And in terms of where it's pulling that hash from that has been signed by keys that are you have a sort of train of trust chain of trust that can give you confidence that that hash is accurate, right? That that hasn't been compromised as well. So it gives you a way to be confident that the packages you're downloading and installing are the latest version, that they haven't been compromised, that it's what you actually asked for and not something else that's being served to you etc.. So there's kind of two broadly two components to it. One is a TUF enabled client, so in this case it's an extension that Ted and Adam wrote to composer that when composer is downloading the various packages, it's going to be checking regularly for each of these each of these packages that it is the package that it hasn't been compromised, etc.. The other side of it is the server side that is integrated into the packaging pipeline. So we've been working with the drupal.org packaging pipeline infrastructure people. Right? And so whenever you go to release a new version of your module, it gets picked up and run through a process that packages it into a zip file, adds the license.txt, various other things of that nature. Right?

Matt Kleve: All the fun stufff project management - module does. Right?

Christopher Gervais: Exactly. Exactly.

Ted Bowman: Yeah.

Christopher Gervais: So what this does is basically take the output of that and then generates all the signatures that are required for the client to then be able to to do all the validation that it's going to do, and then that gets published to to, you know, a static repository that composer then pulls in as need be to to keep up to date on where all of that works and make sure it has all the data that it needs to to be able to do the validation that the client needs to. So that's basically where it's at. We're in the process of working with the drupal.org infrastructure partners to get this deployed sort of in parallel to the packaging pipelines and stufff like that. And so that's kind of where that's at. We're, we're putting some final touches on it. And that's kind of where we should be seeing it deployed to a beta in the next few weeks or months.

Matt Kleve: And if I heard you right, TUF is not Drupal specific. It's something that's been out in the wild for a while, right? Before Drupal. And it's being used otherwise?

Christopher Gervais: Ah, yes.

Matt Kleve: In a similar manner? Like, so it's not - yeah.

Christopher Gervais: Sure. Yeah it is. It's not something that's Drupal specific at all. It is agnostic to the content. So it isn't even only about updates. It can be used for a number of things. It's sort of being led by people in Python community largely. One of the things we did early on with Tim and some of the rest of the team was kind of figure out what our requirements were, what options were available in terms of in terms of server software that we could use to to do the tough deployment. We discovered there wasn't really a choice out there that really worked for us. So we ended up implementing it ourselves and writing our own server for it. One of the primary goals in doing this was to make that the work that we've been doing, open source so that ideally packagist.org, for example, might be able to adopt it and other other package repositories outside of PHP even. So we built it in the context for this particular auto updates capability. But we've ended up splitting off that component into its own sort of project. It's called Rugged. Just play on TUF, right? And so that's sort of a standalone project that we're in the process of sort of standing up and making available to some other communities to see if it'll fit their, their needs as well, in terms of implementing TUF for package management and other, you know, other projects or even in sort of like a private context, right? If, if you know, some, some organizations, governments and so forth need to be running from trusted package repositories and stufff. So it would potentially enable that, you know, those kind of things. But that's all kind of down the road. At this point, we're in the process of sort of finalizing that, putting a beta service so that the client component that Ted was referring to that the auto updates component speaks to, can then do the automated updates truly sort of hands off. Right? So to extend the the, the dishwasher metaphor far past where it makes sense to it would be a way to make sure that the plates you're taking out of the package of the the dishwasher are the same ones you put in. Right? Like that somebody else snuck in and replaced them with, like, I don't know, something else. So that's kind of where that stands.

Matt Kleve: If someone wants to update my dishes, I'm not going to be upset.

[Laughter]

Matt Kleve: If somebody like replaced them with broken ones. I might be mad, I suppose so.

Christopher Gervais: Well, and that's that's where it comes to.

Matt Kleve: Yeah.

Tim Lehnen: Yeah, exactly. I want to I want to amplify a little bit of what Christopher said, because first of all, the work that he and the consensus team did to produce Rugged is really pretty incredible. And I think he's underselling it a little bit in terms of what it could be. You know, TUF, in addition to being pioneered by the Python community, I think it's it's formally adopted as one of the Cloud Native Computing Foundation projects. And it's also collaborated on by the Open Source Security Foundation, which is another sort of working group that is has a number of folks together from all sorts of industries working on software security kinds of things. So there's some really there's some really smart folks kind of looking at the problem space. The whole idea, right, is to not have to depend on, right, you don't have to depend on kind of certificate authorities. You don't have to depend on any of these other things. It's this chain of trust that gets implemented, is secure in and of itself. And, you know, we've already collaborated with other projects who might be interested in in implementing similar things. We had some folks from the Joomla project from Typo3, and certainly plenty of folks outside kind of the PHP world as well. So hopefully we can make the whole web more secure. And Drupal can be one of the first to kind of demonstrate this in the context of an open source project, and that would be great.

Matt Kleve: Outstanding. We're talking automatic updates with some of the folks who helped make that happen. It's a contrib module for now. Here, in a minute, we'll talk a little bit about what's going to happen in the future and dive in a little bit deeper, coming up right after this.

[Intro music].

Matt Kleve: Welcome back. You're listening to the Lullabot podcast. We're talking automatic updates right now in Drupal contrib maybe on its way to core. So during the break, we were talking a little bit about how some folks were speaking at Drupal Con 2021 about securing PHP package repositories with TUF. So there's a link in our show notes to that presentation. So...

Tim Lehnen: Feel free to check it out.

Ted Bowman: Yeah, yeah.

Matt Kleve: [Laughing] When thinking about this automatic update thing, I think the basic site, it would make a lot of sense that set it and forget it. Ted, something in my mind, we have a friend who runs a lawn care website and brags that he doesn't have to mow his lawn ever again because he runs the the website for the guy that runs a lawn care business. And it's the kind of thing that he can, you know, do tweaks every once in a while and update Drupal. And it sounds like it would be the kind of like the perfect thing to be able to have automatic updates and keep Drupal up to date, keep the site safe, and

Ted Bowman: Yeah.

Matt Kleve: Very little effort involved. But maybe there's some some things that don't work great with automatic updates or what wouldn't be a good?

Ted Bowman: Yeah. Yeah. I guess when you wouldn't want to use it is when you have, you know, Drupal specific hosting or hosting where you write protect obviously production. You know, by just by the nature of trying to update something, you have to write to the file system. So that is going to cut out, you know, directly using it on production in the big Drupal hosting ones. On an Acquia and Pantheon is not going to support directly, though I think both the companies, you know, have other solutions for updating.

Matt Kleve: Mm-hmm.

Ted Bowman: So yeah, so that would be the, the main I wouldn't say any particular type of project can't use it. It's just more sort of depending on the, on the system you choose to host on. And and where you choose to use it, you know, do you choose to use it like in a develop like a, maybe a staging environment, maybe something like tugboat where you, you know, you update it, then say, okay, you know, we hit the button, we update it, or maybe we you know, create an environment and triggered an update, you know, client, please look at this to make sure we're we're good. Yeah. So I wouldn't say any particular like, industry or something shouldn't use it. It's just a matter of you know, do you do you already have this solved? Say, like you already have a system, maybe for updates and you're composer experts and you have dev, stage, prod and everything's triggering currently. You know, if you're not seeing this as, oh, this is definitely filling a hole that, that we don't have handled in our current organization. Then then maybe it's not for you, or maybe the whole part of you is...

Matt Kleve: I've seen very few Composer experts. Like...

Ted Bowman: Yes.

Matt Kleve: You might use composer on a regular basis, but are you really a composer expert? I don't know.

Ted Bowman: And I think as... Yeah, as the project expands then we're sort of looking to solve larger sort of composer, at least composer pain points for people. So, you know, you might use this even locally to update your eventually contrib and core. I mean, you could use it for contrib and core. Now, if you're okay with using the experimental module locally. So then you wouldn't have to use composer directly to, to to update everything, but then maybe you would use your version control still to get everything up to production. So if you're using version control to push stufff up or or an artifact, but you still might run the updates locally to push that up there, then you could use that in the workflow there.

Matt Kleve: I think it's important to to say what you said again: a lot of Drupal sites are hosted in Acquia and Pantheon, and so this this would not work there based on how they have things setup.

Ted Bowman: Yes. But in Acq- I know for the Acquia case, we have a cloud IDE where you can actually spin up an environment that is writable.

Matt Kleve: Yep.

Ted Bowman: So and a lot of people use that would use that for development, that's writable code, and then you commit it and then you push it back to the environment. So you could potentially spin up a cloud IDE, run the auto updater, check everything's okay, push it back to an environment either production or another environment, have people check it out, and then go forward from there.

Matt Kleve: Gotcha.

Ted Bowman: So anytime you have a writeable environment and in our case in Acquia we have the cloud IDE, you could use it. So whether... Yeah, go ahead.

Tim Lehnen: From having talked to both platform providers and some others, I, I think there are some latent plans in place to build things into the platform. Like we were talking earlier. There's extensibility. There's there's events you can subscribe to. So I suspect that these platforms are going to start building in some update functionality that becomes more integrated with that full platform. Maybe you're not using it exactly as it comes off the shelf in core, but using a an extended version of that within those platforms.

Matt Kleve: Sure.

Christopher Gervais: And there's sorry, there's there's also some site and project specific things that might be red flags as well, like if you're using an alpha version of a project or something, which, you know, nobody recommends. But if that's the case, the developers may feel comfortable changing the API or something like that. And if you're extending that, you might find that you know it deploying a new a new version of that project, breaks stufff on your site, right? So there's that kind of stufff if you have a lot of custom code that's, you know, leveraging API's and contrib modules, those are things that I think you just would probably want to be a little bit more careful about if only because they it depends on the professionalism and responsibility that the contrib module maintainers are taking in terms of the stability of their their code. Right? Because if you're if you're updating from one version to another and they make some change that they might not even recognize as an API change those are the kind of things that we would probably want to be careful about.

Tim Lehnen: Yeah. And that's a really good point, is that we need to continue fostering good hygiene in the from our maintainers in the community. Good, you know, good understanding of how to use semantic versioning to indicate to properly indicate, yes, this was only a patch release, it didn't change APIs, it didn't break backwards compatibility. And therefore it's safe to automatically update those sorts of things. And I think so far one thing we haven't really talked about is people have tried this out already. People have already started using it since it went stable. And you've had some good success, right, Ted? You've had a number of people giving it a try.

Ted Bowman: Yeah, people are giving it a try. We also had a lot of people at Drupal Con running beta tests on their system and doing reports, and we'll have in the show notes and on the project page, there'll be a sort of report, your success or problems with it. Because because of the people that potentially we're targeting, not everybody is going to feel comfortable going out and making a drupal.org issue to say, hey, I have this problem. And so we're going to have a, have a forum where you can say, you know, it worked or it didn't work. And here is my hosting environment as far as I know. Sort of prompt them to sort of we can get as much information because really we want to get this into core, but we do want to have a sort of record of it being successful for people. Right? We don't want it to release it into core and not have an idea that people are already having success with it. And because of the nature of it being, you know, targeted to help people, you know, keep their site secure, we want want to be really careful when we add it to core, that it's already something that's been tested a lot. So we'll have the note for we'll have it in the show notes and on the project page to say, you know, hey, it was it worked for me or it didn't work for me because a lot of times also, you know, traditionally if if something like this works for you, you're not going to go back and report, right? Because it just works. There's no reason for you to make an issue on Drupal.org and say, thanks everybody. It works great.

Matt Kleve: Yeah.

Ted Bowman: But that would be great, you know? But but generally we want to. And also because it is, you know, because there's not much and hopefully in the future less user interaction than there's not as much for people to say, oh, I'd like this to have a different user experience when I do this or that. I mean, we're going to have a user experience for the settings and for getting it up and going. But once it's fully automated and a lot of the next steps for the contrib module is to enable that fully automated step, which sort of the security team sort of deemed like, let's not do the fully automated step until we have the update framework completed that we can use. But once it's fully automated, then, you know, if it's working for you, we want to know just, just, you know, give a thumbs up and say, yeah, it's working well.

Matt Kleve: Which is something that can be enabled? Or is that am I misunderstood? Is can if I installed the automatic updates module, could I change a flag or.

Ted Bowman: No.

Matt Kleve: No? Okay.

Ted Bowman: We are. Yeah. You can't do that until the update framework stufff is done.

Matt Kleve: Okay.

Ted Bowman: Not not because of any technical reason on the project itself. It's just more it was deemed like not responsible to put that out there until we have the fully the full sort of system that Christopher was talking about.

Matt Kleve: Okay.

Ted Bowman: And this sort of gets you like above and beyond what composer would offer, you know, with the checking of the hashes and stuff like that. It protects against a lot of different things.

Matt Kleve: Very good. And, you know, with automatic updates being stable, I mean, it's good to know that stable releases are covered by the security advisory policy.

Ted Bowman: Yeah.

Matt Kleve: So if there are issues that are need to be solved, it's something that everybody will know about as a part of a security advisory. Right?

Ted Bowman: Yeah.

Tim Lehnen: Absolutely.

Matt Kleve: And the security team will be a part of that. That'll be good.

Ted Bowman: Yeah.

Tim Lehnen: I think one of the other things that would be, you know, we've talked a lot about the scope of what exists in in the the initiative in the contrib module in its current form and the ambitions for what it can do later. Right? We're doing core updates. There's the ambition to add contrib updates and to kind of further extend that functionality, make it more and more automated, make it more and more set and forget. To get there, there's there's a few things that need to happen, right? To get to core, as we've talked about, we had to reach this sort of stable release for the contrib ecosystem, we do need to get the full update framework system integrated into the Drupal.org packaging pipeline which is work that's coming soon, we will need people out there to go ahead and test that stable module in its current form, and then also as the in its fully automated form once that security stuff comes online. But we're close. We're really getting quite close to being able to ship it with core.

Matt Kleve: So now I'm guessing that a lot of this whole, like, secured way to download and update modules or core or that kind of thing, is kind of a similar set of mechanics than what would be used then what would be required for the project browser initiative that's also being worked on. Is there? Tell me about the relationship between automatic updates and the project browser.

Ted Bowman: Yeah. So right now the automatic updates has a submodule called Package Manager, which really just deals with doing these composer operations in a staged way that I described at the beginning of the podcast. So it doesn't really have any idea that it's only for updates, you know, only for updating things on your site. It's just running composer operations. So we are working with the project browser team. Right now, there's, is a stable release for Project Browser? That lets...

Tim Lehnen: At least a beta or even an RC. They're not quite stable yet.

Ted Bowman: Yeah. So they have a browsing ability. And then when you want to select a module, it'll show you how to do it via composer but it doesn't actually install it via composer. But we're working with them to make that use our submodule, which would be then in core eventually to actually do the composer install of the module. So that you would just not, you know, not that you would get instructions, it would just be on your site. So it would use sort of the same mechanics under the hood that we use that we would eventually spin off into its own core module. The only reason it's part of automatic updates right now is just sort of easier for development and we were the first user of it. So yeah. So it would, you know, it's going to enable the project browser to actually be a project browser slash installer. And then anybody, you know, other things we maybe haven't thought of that people might want to do that sort of automated composer operations package manager will provide an API for that. And because it it's one sort of submodule API module that both automatic updates and Project Browser are going to use is that they can they'll be aware of each other. Like you won't be running a core update and having somebody install a new module at the same time. So the fact that it's one module is going to provide a lot of the validation as far as like you have the right version of composer, your file system meets all the requirements will be in one place that is shared by both and that we sort of know, okay, you can't be you can't be running a core update, installing a module, and then using some other contrib module that that does composer operations at the same time. So it keeps sort of keeps them safe.

Tim Lehnen: Yeah. And I think it's a super critical thing that we've kind of glossed over throughout this whole description. Right? Is under the hood, what auto updates is doing, what its package manager is doing, what project browser will be doing, is sort of building and modifying your code base in standard ways, with standard composer operations such that, you know, it's not like if you choose to manage your site with automatic updates, or if you choose to use the project browser, suddenly you are in a, you know, the playmobile version of Drupal and you can't break out of it again, because you've put up these kind of special guardrails and it's actually now a different code base or, you know, installed in incompatible ways. No, it's exactly the same, regardless of whether you had done it all manually or you had done it doing these tools which means, you know, we don't wind up putting people down these sort of tangential paths that might leave them with some sort of dead end if they do need to, like, level up their their kind of DevOps practices, right? You know, if you've got this initial site release from some small organization and they're using they use auto updates, they use Project Browser, they get everything off the ground and then they blow up and get big and and need to, you know, work with an agency out there or someone to, to put in place some more robust practices. They're not going to have to undo some, some, you know, hacked work that was put in place. It's going to work exactly as it should, so.

Matt Kleve: So we've heard with a lot of the other initiatives that are trying to get into core, the the process that that goes through, it has to become an experimental module. And then it kind of goes through the process and then is a part of core! And I'm sure along the lines, there's a discussion of when it's enabled as a part of core, you know, what type of install profiles it would be used in, and that kind of thing. I take it that this is the path we're going down? The idea is to to move to core, right?

Ted Bowman: Yeah. So the initial yeah, I don't think we'd get ten zero right now, but maybe ten one.

Matt Kleve: Okay.

Ted Bowman: And yeah, hopefully get into core...

Matt Kleve: Which is a regular timing for that now. Right? Like we can kind of estimate when that is?

Tim Lehnen: So.

Ted Bowman: Yes. Did not write.

Tim Lehnen: Ten one I believe would be a June release date.

Matt Kleve: Okay.

Tim Lehnen: Yeah. For stable because ten zero should be December.

Matt Kleve: I had to crane my neck and look at the calendar there. So we're talking June of '23, maybe.

Ted Bowman: Yeah.

Matt Kleve: Is the idea.

Tim Lehnen: For a full stable release if it doesn't get to zero, which is not totally out of the question.

Matt Kleve: Okay.

Ted Bowman: Yeah. It would be experimental in core.

Matt Kleve: So it would be the end of '22 or in '23 is kind of the the goal there.

Ted Bowman: Yeah.

Matt Kleve: Okay.

Ted Bowman: But if you want this functionality now definitely we encourage you to use the contrib module. Because we're developing it as we actually have a script that converts it into what would be the core merge request.

Matt Kleve: Okay.

Ted Bowman: So it's not like we're writing it now and then oh, we'll see if it works and we'll rewrite it. I mean, obviously anything can change on the way to core and we'll get more feedback. But the ideas were, you know, we have a we have the core standards being enforced on our contrib module. So the idea is we're trying to develop it now, as it were, for the core module. And everything that's in the main module is the MVP for core. So it's not like you would enable this module that, you know, has XYZ feature, then when you get into core it's like, oh my gosh, I really loved this feature in the contrib module, I wish it was in core. We're trying to sort of have this be the testing ground for the core experience. And core would probably, you know, start with right now it supports just patch releases, but there's a hidden like a config you can set that you can turn on to do minor releases too. It doesn't do major releases of core, but I think the idea probably would be you know, depending on how it goes, patch releases first in core, then minor releases and then maybe contrib after that because there's a lot like I think Tim was saying earlier or Christopher was saying there's a lot to update in contrib. That's not just a problem with the updater. There's a problem with you know, you never you have to be very careful using if you use 50 modules, do those 50 developers have the same idea of what a breaking change is? Yeah.

Christopher Gervais: There's going to be lots of opportunities to improve this over time too, right? Like like Ted just mentioned, this is an MVP, right? There's projects like Upgrade Status that kind of tracks what modules are compatible with the next major version of Drupal and things of that nature that, you know, could be the kind of thing that gets hooked into this at some point. You know, if somebody wants to, to look at that kind of thing. You know, the, the issue we were discussing earlier about making it a writable file system, you know, all that takes is an external something outside of the the web process to trigger the process, to make it so that it doesn't have to be the web user that's writing to the file system. So that could be cron, that could be something in our control panel that either changes the the file ownership or something like that temporarily. There's there's a lot of that kind of stuff that could also come along and sort of like improve that, the security, the, the, the usability the integration with a lot of other projects that are you know, I'm just brainstorming here about some ideas. It's not like these are on the roadmap necessarily, but it provides a framework for us to be able to start really looking at how to integrate some of that stuff into sort of an automated way to keep Drupal up to date without, you know, as light touch as possible.

Matt Kleve: And I'm, I'm willing to bet that the group of people who got this going is much larger than you three who are on this podcast.

Ted Bowman: It was just, just us.

[Laughter]

Ted Bowman: Yeah. Yeah, a lot of people helped along the way recently. You know, we had a lot of beta testers at Drupal Con, and we had people outside of Drupal Con filing beta testing results. On our team, we have a team dedicated at the Drupal Acceleration team. Adam, as somebody mentioned earlier, just shout out to...

Tim Lehnen: He's phenaproxima on Drupal.org.

Ted Bowman: Yeah, yeah. Theresa. Theresa, Yash, Omkar, Rahul, Travis did a lot of the composer stage stuff that actually is like a separate library does the whole staging thing. Yeah. And there were people before, like the you know, the 1.0 version that did the or the 1.x version that did the tarball sites for Drupal seven and Drupal eight. You know, we learned a lot from them. Which I'm...

Tim Lehnen: Yeah. Lucas Hedding did some of that work.

Ted Bowman: Yeah.

Tim Lehnen: Yeah. You know, we've also collaborated with people on the security team. So folks like David Strauss, Michael Hess, Peter Wolanin, Exjam, core, all sorts of folks. And, and Christopher, there were some more folks on the consensus team who helped out as well, right?

Christopher Gervais: Indeed. One of my partners, Derek was instrumental early on in coming up with the architecture and sort of porting some of the, the components. And of course, there's also Neil Drumm on the infrastructure team, right. The D.org infrastructure team that's been key in sort of coordinating some of this stuff and especially the the deployment planning and everything that we're working on these days to kind of bring it to fruition.

Matt Kleve: Right on. Tim do you have anything you'd like to say as we kind of point towards wrapping this up?

Tim Lehnen: Sure. Yeah. I'd really like to just kind of speak direct to the audience for a moment, which is to say that, you know, this is an open source initiative for an open source project. And like everything in Drupal and everything open source, it's only possible because people band together to contribute, right? And there are ways that those of you listening can contribute. You can contribute by trying out Auto Updates and filling out that feedback form to give Ted the information he needs to know, you know, how to polish this and get it ready for core. You can support the Drupal Association, right? You can be a Drupal Association member, you can come to Drupal Con and come to our sessions about these features and others. You can be an organization and do what Lullabot has done, for example, and become one of the supporting partners of the Drupal Association, something we thank you all for very much. And you can also support The Rugged Project. So Christopher at consensus, as he said, has open sourced the Rugged Project. And this is really designed to be a kind of reference server implementation for TUF that could be used absolutely outside of Drupal or even PHP and help secure a supply chain so you can contribute by being part of that project. If you're interested in that, from a kind of code contribution point of view. Or for any larger organizations out there listening, if your organization has an interest in the security of the open web, we could really use sort of support and sponsorship as well. I think that would be good for the ecosystem and the web as a whole. So anyone listening, a bunch of ways to get involved and we'll throw some links in too and we appreciate your help.

Matt Kleve: Christopher, any final thoughts?

Christopher Gervais: Yeah. I mean, basically I would encourage people who are interested in the security elements of it to check out that blog post because there are a lot of complex topics involved in the implementation side. I'm always happy to answer questions about them or whatnot. So feel free to reach out if you do have any questions on on that side. My contact info is on, on the site. But otherwise it's been it's been been really great working with the DA on this project. I look forward to continuing to do so. It's been an interesting problem space to to delve into and learn about because God knows, I know a lot about deploying secure Drupal sites, but the TUF implementation is a whole new world of complexity around how to how to make sure that it can be secured. So I've enjoyed it tremendously. And I'm really glad to see that this is coming to the point where we're going to see see it used and hopefully secure a bunch of sites on an ongoing basis, making it a lot easier for people to use Drupal without necessarily needing to get to the sort of getting into the, the systems engineering side of things that that some of us have had to in our careers.

Matt Kleve: Very good. Ted?

Ted Bowman: We have the slack channel on Drupal slack, hashtag auto updates, and on there you can find out about our bi weekly meetings, which are on Tuesdays. And they're asynchronous, so if you can't be there at that particular time, you know, you can respond or create that, create threads afterwards. We have the issue queue for the auto updates module. So you can file issues. You can help with issues that are already created. So yeah, we definitely want people to test it out. And we're, you know, encouraging anybody who is interested in helping out to join slack and join the issue queue. And now we're sort of spreading into the project browser space. So anybody who's maybe more interested in Project Browser can sort of help us sort of implement what we have in Auto Updates and make Project Browser a composer, you know, composer native operations too. So.

Matt Kleve: And, Ted, you had mentioned earlier that you were hoping to have a feedback form on that Automatic Updates project page.

Ted Bowman: Yep.

Matt Kleve: So if somebody didn't want to go into the issue queue and talk about how their site blew up, they can reach out through that feedback form.

Ted Bowman: Yeah. And it's also for people whose site didn't break [laughing].

Matt Kleve: That's correct. Like, hey, it worked great. Like that's that's good feedback too.

Ted Bowman: Yeah.

Matt Kleve: Hey it's wonderful!

Ted Bowman: I think especially because like, I think we're much more system dependent than a lot of initiatives in Drupal core, right? Or that are going for Drupal core. You know, Layout Builder probably doesn't care so much like what server you're running on and what version of composer you have or your file system. So you know, we're doing new things, so we want to sort of we can only, you know, put so many tests in as far as, you know, it's it's a whole different thing to actually having people running it on different hosting systems.

Speaker6: Yeah, it's something that I thought about but never said was that Drupal can be run in so many different ways. Like you're never quite sure how somebody set up Drupal, and there might be some weird edge case that you haven't thought about yet and maybe need to be integrated.

Ted Bowman: And we may support it. We may not, but we also want to like the user if we're not supporting it, to have really good feedback as to like, you know, we don't support this for X reason or whatever, not just, oh, you just see some really long stack trace on it. So the more people we can sort of get feedback as to why it worked great or it didn't work, we can sort of make the user experience better for everybody running it.

Matt Kleve: Well, great. Thank you for joining me!

Ted Bowman: Yeah. Thanks.

Christopher Gervais: Thank you Matt.

Tim Lehnen: Thanks, Matt. Great to be here.

[Intro music]