Talking Continuous Integration

Transcript

Matt Kleve:

For July 18th, 2019, it's the Lullabot Podcast! Hey, everybody it's the Lullabot Podcast, episode 238. I'm Matt Kleve senior developer at Lullabot. With me as always co-host to the show senior front-end dev Mike Herchel. Hey, Mike.

Mike Herchel:

Hey, good afternoon Matthew Kleve.

Matt Kleve:

Good afternoon Mike. So with this is the Lullabot Podcast we talk about all things Lullabot. Lullabot's strategy design development company working primarily in Drupal.

Mike Herchel:

Yes.

Matt Kleve:

And today we're not talking about Drupal.

Mike Herchel:

No. And we're not talking about strategy design. I guess we're talking a little bit about development.

Matt Kleve:

Development that's right. We're going to talk continuous integration or CI.

Mike Herchel:

Yes. So you're going to hear every ... Like for the audience we're going to be saying the words CI, the letter CI and that stands for Continuous Integration.

Matt Kleve:

Charlie India.

Mike Herchel:

Yup.

Matt Kleve:

Or something.

Mike Herchel:

Something like that. And so we got three CI experts on the call. All of them are Lullabots.

Matt Kleve:

Purported to be Lullabots yeah.

Mike Herchel:

First up, we have a senior technical architect who lives in London, England and has an irrational fear of pine cones. Welcome back Sally Young who is a senior technical architect.

Sally Young:

Wow. I can't believe you told the whole internet about that. So FYI, it was when I was a child. I'm perfectly fine around pine cones these days.

Matt Kleve:

All right.

Mike Herchel:

Also with us today coming from Guelph, Ontario, Canada, we have Andrew Berry, a senior architect at Lullabot. Hey, Andrew?

Andrew Berry:

Hello. I think that is the first time I've heard somebody mispronounced Ontario and not Guelph so thumbs up to you man.

Mike Herchel:

I'm glad I'm here for that.

Matt Kleve:

And lastly but not leastly we have James Sansbury who is a development manager who also tends to do a lot of our continuous integration type tasks he lives outside of Atlanta in Georgia. Welcome James.

James Sansbury:

Yeah, thanks [inaudible 00:02:19]. Glad to be here.

Matt Kleve:

So we're talking continuous integration or CI remember it's something that was probably first added to a project I was working on I don't know, a few years ago here at Lullabot. When did this kind of get started and why did it get going? I remember it was this magical Jenkins things that Jenkins thing that did stuff for us and it was good right?

Sally Young:

Well before we used to have a folder called scripts

Matt Kleve:

Yeah.

Sally Young:

And it used to be full of lots of [inaudible 00:02:49] bash scripts that may or may not run on your local depending on what you had installed that day, and you'd probably have to grab a load of secret passwords to inject them and they're all sitting around on your laptop, and sometimes it will break because you ran something in a different order or you did something differently or you didn't have some environment variable and it was just a recipe for lots of things going wrong. So it's good to make things irrepeatable because they don't get messed up so often and they're a lot easier to debug when something does go bad.

Andrew Berry:

I think the other part that's interesting is if you sort of look at the history of it, it really became more popular as more and more development happened with services that were hosted outside of your company or outside of your office. You go back 20, 30 years in development shops would have their own internal servers literally in their office because they didn't have necessarily great connectivity out to the Internet. You'd be running CVS or some version or source safe or something like that and you might have a build server, but every single company would be doing it on its own setup completely independently. And now that` so much development for those sorts of companies and industries happens on cloud services anyways, the ability to wire up some sort of continuous integration set up, even if it is ending up to be sort of self hosted is way easier than it used to be.

Mike Herchel:

Let's take a step back and maybe even define what continuous integration is before we get too deep into it if that's all right.

Andrew Berry:

You push code and stuff happens. Yeah.

Matt Kleve:

So I think it's... To kind of bring it back to what Sally said, we had this magical set of scripts in a scripts directory that was quite persnickety and might not be perfect for everyone and we make it perfect for everyone by outsourcing all of those tasks to one place.

Sally Young:

Although that was probably even a step too far like before that, I like that Andrew also assumed that all these companies had some kind of version control [inaudible 00:05:00]. It would have been like-

Matt Kleve:

Yeah, it's my file dash two dash this is the right one.PHP.

Sally Young:

Well, to launch the website just follow the instructions that Dave has written on the whiteboard and just stick it on the FTP server and you'll be fine.

Andrew Berry:

No, I mean like if you go back far enough it's kind of like, yeah. I definitely did work on sites where there was like even Drupal sites that had zero version control and you kind of have this evolution from FTP and you're just uploading files and doing it live all the time to sticking it into version control so that you have some record of what you're changing. And even if it's not automated you're still using version control to somehow get your code out to those production sites to then hooking version control up to some other build system or continuous integration systems. So the actions that you would maybe have to do manually, like go to production and check out the latest code and make sure everything works we're automated.

James Sansbury:

Yeah. So like in my particular project that I'm on right now when I'm merge a pull request a process goes that where it triggers a job which will maybe compile my sass into CSS and my ES6 style JavaScript into ES5 style JavaScript and it'll maybe run composer and pull down all these dependencies put everything together. Then it will maybe write, it'll run some Be hat tests or something to make sure things are actually working. And at that point it pushes that whole bundle of code that's, a lot of it is dependencies out that's outside of my repo. It pushes that whole artifact up to a development server which then people can kind of look at.

Mike Herchel:

So, yeah. So what do we need to do to do things like that? I mean, is that, I think it was Sally that mentioned a bunch of bash scripts or maybe make files. I mean, how did it start? I've heard Jenkins, I've heard the word circle, CircleCI, I know that Lullabot has a product called tugboat. What do we need to kind of get started on this?

Sally Young:

Well, so you mentioned Jenkins and that was sort of one of the first popular ones that came out, that was called Hudson back then?

Andrew Berry:

Yeah.

Sally Young:

And that is something that we're self hosted it's a Java project. You could essentially paste your bash scripts into form fields and then have some jobs that you could click and run and yeah. It was pretty well.

Mike Herchel:

Sounds exciting and terrifying at the same time

Sally Young:

Yeah exactly.

Mike Herchel:

Kind of like turning PHP module on a site and saying well, I'll just write my module here.

Sally Young:

So interestingly, I used to write my first CI scripts with this package code thing which was all in like XML and-

Mike Herchel:

P-H-I-N-G right?

Sally Young:

Yes. I think that's like a really-

Mike Herchel:

It's still around.

Sally Young:

Yeah. I'm not surprised. It was good at the time there wasn't like much else out there. Anyway, but like Jenkins had all these kinds of plugins and still does that you can read all these types of files and then it has its own steps that you can add, like it knows about Git and GitHub and Git all of those things and the problem with running it with Jenkins is that you have the most amount of control possible is that you don't have to administrate, administer this service. Say if you're self hosting there's all the problems that come with upgrading and so this is like a maintenance cost to having that. And something goes wrong it's kind of on you to have to fix that.

Matt Kleve:

And Dave's instructions on the whiteboard have kind of turned into the Jenkins server that Dave set up and knows how it works.

Sally Young:

Yeah. Right, exactly. So, a lot of cloud services appeared like CircleCI, Travis CI. Are they still around?

Andrew Berry:

Travis still is [inaudible 00:09:07] some other company and there is some big kerfuffle about-

Sally Young:

Yeah.

Andrew Berry:

Them.

Sally Young:

A code ship, beanstalk on AWS it was like a billion of them now yeah. So, they take a lot of maintenance costs away allegedly though, obviously because they have their own ways of doing things. As you do get problems with each one of those and each one does it slightly differently and not.

Mike Herchel:

So is now a good time to maybe talk about some of the ones that we like to use and maybe some of the drawbacks of, you mentioned a drawback with Jenkins is that you have to host it and upgrade it yourself or what is the drawback with say CircleCI and code ship or something like that besides maybe costs and things like that?

Sally Young:

Costs.

Mike Herchel:

Yeah-

Sally Young:

I love Cycle CI use that for everything now. I'm trying to think what drove... So it can be kind of tricky if you want to test a new job. So a job is a set of tasks that you want to do on your CI. And if you're trying to do and use, like make a new set of steps that are going to run on this cloud service testing it can be really frustrating because if something goes wrong, sometimes you have to start [inaudible 00:10:33], run the job from the beginning, get to a point where you think you fixed it and kind of keeps going down. CircleCI does have some tools to help with that so that you can SSH into containers. There's like a local command line tool as well where you can run some types of jobs locally, not all of them and sometimes it doesn't work in seconds of circumstances, but yeah, it does make it a little easier, but it can be a really, frustrating and slow process to get these jobs actually running in the way you want.

Andrew Berry:

The thing that we found, so I've used CircleCI ton on the past three or four projects I've been on. And I was actually on a project where there was originally a totally custom built Jenkins server that the client had set up. And when we looked at the time put in from mean hitting and even then CircleCIs costs were trivial compared to the hours that they'd put into mean hitting their own Jenkins instance. And that might be different for your organization if there's another team who's already mean hitting just a mean hitting Jenkins for you then maybe that's the cheapest option.

Andrew Berry:

But as far as... I would say there is obviously the cost factor with something like a CircleCI that may or may not matter to your organization. The other side of it is if you're using it as a really deep and integrated part of your development process when they go down and inevitably they will have downtime, you can't do anything to fix it most of the time like you're just waiting on their team. And sometimes that can be a hard message to send to stakeholders. Sometimes it's easier just it depends on your team. I've worked with teams where they would much be much happier to know that their own team can be responsible for things fixing things.

Matt Kleve:

For better for worse though. Right. It's like it's your problem or it's not your problem, which would you prefer?

Andrew Berry:

Right, exactly-

Sally Young:

But then also like if you've hosted it you're probably not hosting it in your office anymore and so you still have the same problem if you host it on AWS and that's a problem with AWS, you still can't fix it such as the cloud world we live in these days.

Andrew Berry:

Exactly-

Matt Kleve:

But you've duplicated across a couple of regions right? On AWS you've done it right?

Andrew Berry:

Man, everyone says they're going to do that and no one actually like I've heard from... If you look at your own sites or your client's sites and just ask yourself how many of their production critical or sort of critical websites or applications are actually hosted in multiple regions. And in the case of even like Drupal enterprise sites very few of them ever are so what are the chances if you don't have the engineering bandwidth to do that for the thing that you're actually building, that you're then going to do that for your build server environments with Jenkins or something else? Most of the time when we look at Jenkins servers that have already been set up it's very much sort of the old school single big server with 64 gigs of RAM and a whole bunch of jobs on it. Instead of a hundred little tiny build nodes that if one of them goes down it doesn't really matter. There just tends to not be the the focus on resiliency that way.

Mike Herchel:

It's interesting that we've talked about this directory of scripts that turned into this nice task runner that does everything for us and now it's mission critical. Like we've kind of made that leap into downtime in our CI system means we're down. When really it was started out being this is a nice thing to have.

Andrew Berry:

Well, it means you're down from a, probably from a deployment and development standpoint, like if you're using CI to deploy to production and you had one scheduled and then there's an outage of any kind, then you can't do it at least not easily-

Mike Herchel:

Manually it's still possible right? Because if you can program the CI system to do it, you could push through it if you needed to right?

Andrew Berry:

Yeah. It's just a question of how much time it takes to do that. And especially if you've got multiple systems that need to be in sync. As soon as you've got a person doing it by hand there's a lot more room for error.

Mike Herchel:

I think it's also important to point out when we gave our early examples about what people are using CI for I don't think we mentioned deployments. So Andrew you kind of jumped into, using a CI system for deployments. How would that work and what would that look like?

Andrew Berry:

Yeah. The most recent example I have of that is you've got your Drupal site and masters, your primary development branch and then you're ready to push something in production. So someone creates a tag and that goes through and automatically does all the build processes meaning, composer install and NPM and all those other things. And then it would usually stop and someone would say, "Okay, I am ready to actually push this to production now. I'm going to click the single button that resumed the workflow or resumes the job and sends it all out." And it's Drupal sites in particular we tend to do that because a lot of updates involve a tiny bit of downtime in terms of database updates and those sorts of things. So you don't necessarily want it to be totally, I'm not going to say random, but sort of at the mercies of how long your jobs takes to execute to determine when that process actually happens.

Mike Herchel:

Sure.

Sally Young:

Then there's another sign to deploys where like the target that you're going to doesn't necessarily have all the things it needs yet. So you need a build step like in Drupal you need to run, compose or install maybe you want to remove test files or other sensitive files that you don't want to be out there.

Andrew Berry:

Yeah. I think the other thing that's really interesting for me like weighing the pros and cons standpoint is that, at least for me having the efficiency gains that you get from just your day to day development of having a really finely tuned continuous integration set up far outweighs the downsides of downtime or even maintenance of something like Jenkins totally on your own. Like I'd still rather have it than not and maybe actually maybe Sally you could, it'd be interesting to hear from you like what from a day to day development standpoint, what happens as far as continuous integration workflow goes and how it changes the way you work.

Sally Young:

So one big thing is it saves us lots of time, because it has really brought, so most of my experience these days is with CircleCI, or have you said this? So I'll just kind of talk about it in the context of CircleCI. It has really tight integrations with GitHub. So anytime we were running tests and stuff like that we can actually get it to generate all these really really nice reports which you can get posted back to this new API to get how much has housing has actually so a little display kind of in one place. So I didn't have to go in and be like, Hey by the way the test failed. Things like that so that feedback loops a lot quicker. In terms of deploying things we can be quite hands off for a lot of things. So like every time we make pull requests we use our CI to spin up environments. In Pantheon in the current project one and on hierarchy so-

Mike Herchel:

What is RCI?

Sally Young:

What do you mean?

Mike Herchel:

You said the word, the letters RCI. [crosstalk 00:18:20]. Oh, our CI. Geez. All right, strike that from the record. I'm sorry.

Sally Young:

It's okay. In the CI we're using on my current project we're using it to spin up environments for all of our pull requests as well which is really helpful for testing and also making sure that stakeholders can see what we've built is what they actually wanted. In our case because the site I'm working on right now has decoupled on and It's got a bunch of services. This is really great because being able to test stuff locally, quickly, sometimes can be a pain if someone's not doing development day to day. So they can just click through and see all those environments which is really good. And then we also have automatic deploys to staging environments for example. So everyone could always see your latest stuff that's going on and we just don't have to think about any of that stuff anymore now it's all sort of in there and done, which is really good saves me a lot of time.

Andrew Berry:

Back on the one thing you mentioned Sally about being integrated with GitHub and providing metrics about... You could have various tests and stuff like that. One interesting side effect of that is on the emotional side of things for developers, we can have an agreed upon set of standards and we can write, we can program the tools to report on those standards and it really takes like this personal aspect of peer review, you know? Whereas previously it would be maybe the architect or someone else coming in and saying, oh, this is not... This doesn't follow Drupal coding standards here this needs to be updated and it can feel those sorts of conversations can be tedious and they can also where on the developers themselves thinking like I'm constantly being criticized by my architect or the lead developer whoever it might be in and having software do that and just and provide that feedback really takes sort of a emotional burden I think off of the project itself.

Sally Young:

Yes. I could not agree more.

Matt Kleve:

I never heard of it its the first time but it makes a lot of sense.

Sally Young:

There's a tool we use on our JavaScript projects called Prettier which just makes decisions about all those style things. I don't know if there is, well there's coat sniffer for Drupal we use it as well. And it's got some like extra vigilant standards. We've got [inaudible 00:21:04] on there too, so I think we're getting same kind of stuff but with Prettier when you save the files that can actually fix all the things for you which is really nice.

Andrew Berry:

Yeah. One of the things that we try and do in a lot of our projects is make sure that the tools that CI uses are the same tools that you run locally. And so on the project I'm on it's all PHP code right now but, we basically have the Drupal coding standards and I think we have like PHP master detector and a couple other things like that. But basically we have the configurations committed to the repository so you don't have to do anything. You can just run PHPCS or PHPcbf, which is I imagine like Prettier, where basically re formats your code for you. And-

Matt Kleve:

Wait, what was that last one? I've never used that Andrew.

Andrew Berry:

It comes with PHPCS like its part of-

Matt Kleve:

Is that the thing that says that it will fix it for you if you let it?

Andrew Berry:

Right.

Sally Young:

Yes. The code beautifier.

Andrew Berry:

Exactly.

Matt Kleve:

I don't think I'm, I'm confident enough in letting tools change my code. But does it work well?

Andrew Berry:

Oh, I've never-

Sally Young:

Yeah. Really good.

Andrew Berry:

Yes. The only thing, which is like, let's say you have a requirement that every method has documentation. It's going to put it on the empty common blocks above those methods for you. But obviously it's not going to write the documentation. So as a developer you'll have to run it and then run [inaudible 00:22:27] and see what it actually did and see like okay, I've got to fill in the summary for this. But yeah it's-

Matt Kleve:

It doesn't, it does enough to, to make the code sniffing task shut up?

Andrew Berry:

Not Quite. Oh yeah. We're like in the example of a documentation fix it. We'll add the stubs for you like it will maybe add the parameters and there types and so on. But the code sniffer will still fail unless you've disabled the rule because you haven't put the English descriptions of what you need to fill in.

Sally Young:

There's stuff it can fully fix so like if you use double quotes instead of single quotes in your case style what's one or the other it will fix all that stuff for you.

Andrew Berry:

One of the-

Matt Kleve:

So-

Andrew Berry:

Common ones we run into is when you're doing a bit of refactor or something and you have some use statements up at the top that are no longer actually being used in your code.

Mike Herchel:

Yeah.

Andrew Berry:

And it'll just automate it'll let you know, hey these use statements are still here even though you're not using them anymore. And then the, the code beautifier will actually remove those for you so save some time.

Mike Herchel:

So do you have to configure that or is there maybe like some configuration dot file included the Drupal that automatically will configure that or how does it know tabs versus spaces? And-

Sally Young:

So I think develop Montreal has it all?

Andrew Berry:

It's Coder.

Sally Young:

Coder? Yeah sorry.

Andrew Berry:

And, and Mike it has a plugin. So it's PHPCS which is for like generic PHP stuff and Drupal has a plugin that has the Drupal standards.

Mike Herchel:

Okay.

Sally Young:

But this too you can use.

Andrew Berry:

Correct.

Sally Young:

One is true just Drupal and that's kind of some basic things and then there's another one called drip of practice and that's like really finicky that's so nice.

Andrew Berry:

Drupal practice will do things like if you're calling out to interpolate to the global Drupal object instead of using container inject or dependency injection, it will flag those and say you should not be using that even though it is technically like valid code that works just fine. It'll have more of those sort of best practices there. So I find it works really well for new code bases because then you don't have any technical debt to deal with. It can be a problem if you're pulling it into a code base that has either been sort of I'll say roughly ported in from Drupal seven where it was just like let's just get this working on Drupal eight instead of following all the Drupal eight practices or just for custom code that wasn't following those things in the first place.

James Sansbury:

I also want to state that as all of these terminology is kind of going by me. I'm taking notes of everything and we're going to link everything and maybe add some descriptions for the show notes for everyone.

Matt Kleve:

So let's imagine for a second that Dave with the whiteboard from our previous example is listening and Dave says, "Holy crap. I need to figure out this CircleCI thing." And he goes to the website and he sees that CircleCI has a free tier. How long will that actually last for Dave? Is that-

Sally Young:

The free tier is great because, it gives you one container which is like one instance of a virtual machine that will run and you can use that as much as you want. I think you can even use it across multiple private repositories, so the downside to that is when you start getting more jobs. So CircleCI has this concept of a workflow and you can have multiple jobs in that workflow. Lets say the first part is you build something and then you fan out into three different jobs that test three different services or run different kinds of tests. Then obviously it's going to be a bit slow because you only have one container available to do all of those things. And then that's where they get you on the upgrade. You also get a restricted number of build minutes I think so if you SSH into the container, so there's only a certain amount of time you can stay in there per month.

Andrew Berry:

SSH I think is special in that they limit the job to two hours. Because if you have one container say and you've got a container pause waiting for you to connect into it, it's going to block every other job until you say I'm done with this please destroy this environment.

Sally Young:

I think if you don't connect to any environment within 20 minutes.

Andrew Berry:

Right? Yes. The other thing that, just I always wouldn't go for as a best practice is I really try to do as much as we can as far as the CI set up so that it's CI environment agnostic, meaning I use CircleCIs workflow features and so on.

Andrew Berry:

All their configuration as best I can, but as soon as I can I drop out of the CircleCI configuration file into something like a bash script or a Robo file, which is a PHP task runner like grantor golf if you're coming from the Java Script world. Because then it means if CircleCI changes their model and it's like we're no longer giving free private [inaudible 00:27:19] so anyone or we stop using offering opensource projects free containers, like whatever it is, it's a lot easier to change to a competitor. And I feel like it also means that it makes a lot easier to run those jobs locally, you don't need to be in CircleCI infrastructure to do most of the work. And in our case where we're typically using multiple CI providers like tugboat and CircleCI, we can for this the typical steps of like running Drupal database updates and configuration imports and all of that we can share that without having to worry about only updating in one place.

Mike Herchel:

Yeah, we haven't really gotten away from the scripts directory necessarily, but it's plugging it into these CI tools.

Sally Young:

So one of the things we do is well if this infrastructure is in docker containers, so we use it for local development and then it's the same stack that gets spun up inside our test environment. So we know they're running with the same versions of everything and in the same place and using all the exact same script. So if you wanted to run these things yourselves the only thing you could like look through the steps of the cycle CI file, do it manually and the only thing that we've missing would be the environment variables that have been injected in.

Matt Kleve:

So just to be clear on the CircleCI and how stuff works, I see a thousand build minutes per month, if I have tests that take 40 minutes, which I've been on a project that does, does that mean I can only do 25 and remain on the free tier? [crosstalk 00:28:58].

Andrew Berry:

They've reduced that I think it used to be more than a thousand.[inaudible 00:29:03].

Matt Kleve:

Okay.

Andrew Berry:

Because I can't imagine like, because that also means if you have five jobs that run in parallel, like let's just say you have one that's like, typically it's maybe why I say you're paying for it at that point. But if you just imagine you have five jobs like the code sniffer job, your test job and, maybe a Be hat test or something like that, like that's going to be writing more minutes than just, one job by itself.

Matt Kleve:

And I'm sure they're paying for their processing, time and their cloud power, and so it makes sense that they need to pass that on. I mean, you can't give everything away.

Andrew Berry:

You do run into this balance of like how far do you optimize your CI jobs? The first time you do something you might discover that your tests take 20 minutes to run for whatever reason. And is it worth six hours of development time to make it so that they can run the entire 10 minutes? Maybe? I mean it depends how big your team is and how much development churn you have and so on. And what your costs are to just throw more money at your CI provider for more parallel jobs.But there is definitely like I've worked on projects where the first pass of say a migration job was hitting, a general time at lik, it was taking longer than five hours. So CircleCI was just like, no! This is not happening and you know it then made sense to spend some time optimizing that.

Andrew Berry:

The other thing we typically do is try and do sort of fan out configurations. And you can do this with any CI provider where for example, maybe you've got three different jobs using the even simpler example, like two different jobs, one that checks code standards and one that runs a unit tests. They both need composer to run before either one of them can work. And so you have one step which is run, composer install, and then you tell CircleCI okay, there are two new jobs at one for the tests and one for the CI and they're going to use this same starting point as their base. So you're only running composer once instead of twice and obviously you can get a lot more savings depending on how many jobs you have.

Matt Kleve:

Why is circle the one that we've gravitated toward?

Andrew Berry:

See for me it was because as much as docker makes me angry I think it's perfect for CI-

Matt Kleve:

No docker isn't... Okay you're saying that docker works well with CI solutions?

Andrew Berry:

Right.

Matt Kleve:

Okay.

Andrew Berry:

And CircleCI was one of the first that I was aware of where their entire infrastructure was built on docker. So it meant you weren't dealing with a lot of the performance issues you might run into previous solutions. For me also at this point, it's a little bit of inertia like I've been using CircleCI for two, three years now and it has yet to do something that makes me really want to look for something else.

Mike Herchel:

There are some-

Sally Young:

[inaudible 00:32:04].

Mike Herchel:

Of them that we haven't talked about, like, sorry, Sally.

Sally Young:

No, no go on.

Mike Herchel:

There's some other ones that we haven't talked about like a GitLab CI and bit bucket pipelines and those are, if you're using GitLab or bit bucket, those are perfectly viable options as well. One thing, one advantage though to using something like CircleCI is that you are agnostic to whatever repository manager you're using. So there's kind of a catch 22 there where like it's really convenient to use those because they're built into where you're hosting repository. But then if in the future a GitLab make some change to their SLA, that means your company can no longer use it or something like that then you've got to switch everything over to some other CI provider or something like that. [inaudible 00:32:53].

Andrew Berry:

But like I'm a big fan of just using whatever tools you have. Like if you're already on GitLab and you're paying the for GitHub, GitLabs, CIA tools, like just use that until you find a reason not to use it. I've yet to use a CI tool that was, had such issues and was problematic in a way where I was like, well I wish we just didn't you do CI?

Mike Herchel:

Yes, I agree. Makes Sense. I believe gap has what, look out of actions or something like that now, is that correct?

Sally Young:

Well, I think it's still in the Beta and seems to have not got out of the Beta, so-

Mike Herchel:

But Gmail was in Beta for how many years right?

Sally Young:

Yeah. But everyone gets to read the-

Mike Herchel:

Okay, so it isn't working or,

Sally Young:

I don't know. I don't, I don't have access to it.

Mike Herchel:

Oh, okay. So, it's like a private Beta.

Sally Young:

Well, they do that thing where it's like sign up to be on the waiting list. And there are other GitHub services that have launched since that people have signed up for and got in a lot quicker. So I do actually seem to have stalled a bit.

Mike Herchel:

Okay. So when you're writing the scripts like what type of file? Like what type of format are there? Is this like bash scripts? Is this like I know YMAL or-

Sally Young:

I use SH because it's the most portable possible. Even though I'm probably making my life a little bit more difficult than it needs to be.

Mike Herchel:

Why would that make it more difficult? It seems very basic and understandable.

Sally Young:

Well, there's a few things that like bash does a little bit better than SH.

Mike Herchel:

Wait, you mean you literally use SH?

Sally Young:

I do, but it's also just because I am very familiar with it and so and the reason I started using it I think it was because of like all these really slim docker containers you get they just don't have bash.

Andrew Berry:

Yeah, one of my colleagues and I, we maintain a set of starter scripts for Drupal sites. One set is for, I'm building a Drupal site and that has content and database and so on. And then the other is for, I'm building a private module like a Drupal module that is just not public for one reason or another. And in those we do use a tiny bit of bash and then we also have a Robo file, which is PHP. And again I think, I don't know if I could bring myself to use raw SH as far as like how low the features that it doesn't have compared to bash or whatever. But I would definitely say like use the tooling that your team is familiar with. Like the reason we use Robo files on a Drupal site is because you are writing in PHP.

Andrew Berry:

Like there's an expectation that if you're doing CI with Drupal that you know enough PHP to work around that. And I imagine it's the same thing with a Java script project. On some prior tools and sorry, prior projects, I know Sally you have to like we use make files for CI, which I personally think is awesome and I really like how it handles dependency resolution and all that kind of stuff. But the problem you run into it with it is that Oh, there isn't necessarily a lot of experience with make unless you have experience outside of sort of the modern web ecosystem. So then when you're passing it off to some other developer they might not be very happy with what they're inheriting.

Sally Young:

Yeah, make it super cool and everyone could use it. The thing is I think because we're building websites we're not compiling C programs with it actually like the features that you use of maker so small. I think they're actually pretty self explanatory when you open them and I inherited a project that had once I hated it even though I know PHP, I found it impossible to follow and impossible to debug and it was just like, I can't believe I'm sifting through like this giant pile of files. It's eventually just going to run one shell command and it was just ridiculous. I was like why don't you just write shell command for goodness sake? And James will love this because I used to use grunt a lot as well and it's like you'd write all these things with Java scripts and he's like, why didn't you write shell command. And I was like, no, no, no. She'd use it in the language that we're all working in and believe you changed my mind on that. It's just-

Andrew Berry:

What's funny is-

Sally Young:

Huge pain.

Andrew Berry:

I've now come around to the other side where,

Sally Young:

No!

Andrew Berry:

Been on so many projects where I've been like, no, we're using make or no, we're using bash scripts or whatever because this is like the most explicit way that we can describe what we're doing here. But then when something goes wrong it's like nobody knows like people get in there and they just don't know what's going on. So on the project we're on now the georgia.gov project we're not using Robo but we are everything is a dress commands which pulls in Robo and you can use all the fun goodies of Robo task running and rollbacks and all that stuff which is pretty slick stuff but I totally agree Sally Leica, when I first looked at Robo stuff I was just like oh my gosh, this would be one line in Shalon. I would understand it but, yeah it took some getting used to and now I'm seeing the benefit of both sides.

Sally Young:

Right? Yeah. There are benefits. Right. I just felt like anytime I hit like a real problem. Ultimately it came down to whatever, exact statement someone had written in some really deep Robo file. I don't know if I had [inaudible 00:38:57] some folks show all of this and now I can't debug it, blah, blah, blah.

Andrew Berry:

The interesting thing though is like, especially with migrations I find there tends to be a lot of sort of just work that has to be done to run everything. And I love the fact that if I need to I can literally debug the Robo file in my ID just like any other PHP command or PHP script. And there was one time-

Sally Young:

Andrew just use S trace command.[inaudible 00:39:32] even joking.

Andrew Berry:

I've definitely used S trace before. The other thing though with something like Robo is if you run it in verbose mode it will dump out every shell command that's being executed ever get commander, composer commanD, whatever it is you're running. And I find that useful as well because sometimes I want to not divide the Robo file but I want to debug the command that that's being run, maybe it's address command to do something specific and it's nice to be able to just[inaudible 00:40:04], say I can copy and paste out and then run from there.

Matt Kleve:

Right on. Hey, we're talking CI on the Lullabot podcast coming up right after this short break. We're going to talk about integration with other environments. Oftentimes Drupal projects deal with Acquia or Pantheon or that kind of thing, and we'll talk about how CI can work with within that set up and we'll also hear from Andrew and Sally and James about what their ideal setup would look like coming up right after this.

Matt Kleve:

Hey, it's Nick from Drupal Camp Colorado. What's happening with this year's camp Nick?

Nick:

We'll be at the King Center on the Array area campus in Downtown Denver, August 2nd through fourth. On Friday we'll have trainings in summits, Saturday we'll have keynotes sessions and then a party to cap the day off and then we'll have mentored sprints on Sunday. As always, registration will be free, but a personal donation of 25 bucks or more gets you this year's custom designed camp tee shirt and good Drupal Karma. Keep an eye on our website for registration and session announcements @drupalcampcolorado.org

Matt Kleve:

Welcome back. We're talking about continuous integration, aka CI or as Sally RCI.

Andrew Berry:

Or the magic things that happen when we want things to happen. Like we want to tag a release and a bunch of magic happens or we want to create a pull request and a bunch of magic happens. All these niceties that developers love.

Matt Kleve:

Yes, so coming back into this let's talk about how we integrate this into maybe some popular Drupal hosting environments.

Andrew Berry:

As it turns out most people don't have their Drupal site on GoDaddy.

Matt Kleve:

Thank God. Yeah. So a couple of popular ones that we've worked with before are Acquia and Pantheon and maybe we can even talk about it. Just kind of like maybe internal servers and maybe that what that would look like. So, who wants to go first and a pickup provider there-

Andrew Berry:

Who's integrated with Acquia before? James, I know that, I think we've been on projects that were Acquia hosted and had great CI stuff working that you may have made happen.

James Sansbury:

Yeah. So I mentioned earlier the georgia.gov project, so that's currently a multi site set up hosted on Acquia cloud, and we're using CircleCI to build up to run all of our automated tests and then also build up the tag two that gets deployed to Acquia. So what that looks like from like the programming side of things is we have, if someone were to go to bit bucket which is where their repository is hosted in clone the repository they would not get the composer vendor directory they wouldn't get Drupal core. We're using the Drupal composer project to build up the site and so none of those like contrib modules anything that's required with composer, any of the node modules that we're using for the theme, none of that stuff is actually in the repository or committed to the repository. And so the first thing when someone pushes up a tag to bit bucket we have a CircleCI job that will listen for that and take that tag and then-

Matt Kleve:

Wait, there's and Acquia. So Acquia is then integrated with bit bucket. Sorry, I missed that.

James Sansbury:

Yeah. So normally or maybe not normally but Acquia does have a Git hosted, you know, that you, you Git URL and you can push directly to there but we're not none of the developers on this project are pushing directly to Acquia is get repository. So instead the repository is hosted on bit bucket and the developers are pushing there and then CircleCI bridges the gap between bit bucket and the Acquia get repository. Does that make sense?

Matt Kleve:

Yeah, that's the gap I wanted to point out. Yes.

James Sansbury:

Yeah. Some of this stuff I start to take for granted because we've been working in such a way for a while so definitely pull those bits out as I forget. But, so yeah we have a CircleCI job that's listening for tags and compiling, so it'll, it'll run composer install and there, with all the production flags that you want for a composer install job to run and then builds up the theme and then force commits all of those to a new tag called, like, so if you had a tag called V1, this particular script would then create a new tag called V one dash compiled and push that up to Acquia and then you could deploy that tag where you see fit on whichever environment and automatically we're also doing that where we push up, we have the Dev environment that has checked out the master dash compiled branch. And so likewise, while we're listening for a tag, we're also listening for commits to the master branch and compiling that up and pushing that up to Acquia.

James Sansbury:

There's some other things that Acquia has, like Acquia has its concept of hooks so when certain things happen Acquia will automatically do other operations. For example, like if you copied a database over or if you updated a tag, you could set up shell scripts in there. So there's some stuff you can kind of get without having any sort of paid CI or even using something it might be built into your repository management so that you can use the Acquia hooks that are built in. They are kind of rudimentary and it feels like working in a black box a little bit so I find having some of the CII stuff there to be helpful.

Mike Herchel:

Can their hooks do you like to compose or integration, renew the render composer install and all that good stuff?

James Sansbury:

I don't believe so. I mean I guess theoretically you could install composer manually on various environments but it doesn't seem like that would be a good best practice to recommend. So I think usually you'd want something to kind of bridge the gap or there is still something to be said for when, if some of that stuff feels, if you're on a small team or if it feels over your head, there's really nothing wrong with committing those vendor directories and node modules to your project. I know it feels can feel dirty or whatever but sometimes you know, it's like that the simplest choice sometimes is the best.

Mike Herchel:

I'm pretty sure with Acquia at least it used to be, and this is years out of date that you don't have right access to your web code if you're logged in?

James Sansbury:

So I don't think the hooks would have permission to actually write the vendor directory and so on.

Mike Herchel:

That makes sense.

James Sansbury:

The other thing I would watch out for is, you know, and if you're on a small team, this probably isn't going to matter as much. But if you're using like a development released modules from drupal.org those are going to be checked out as Git repositories, which means they become sub-modules within your vendor directory. You're committing that, which is really painful to deal with. And if you're in a small team you maybe never have had to deal with that before so it can really throw developers for a loop. So, as long as you sticking to it stable tag releases of Drupal modules it's fine. But just be careful of that edge case because, if you inevitably get bitten by it at some point.

Matt Kleve:

Right on. So not everything is in Acquia. We have options, and Pantheon is another popular platform. Sally, I think you've done some work as far as integrating some great CIA stuff within Pantheon, right?

Sally Young:

Yeah. It's actually quite similar to what you would do with Acquia or any cloud hosts really. So yeah you do run all the composer install stuff, you check in all the vendor directories, it's a fee in this remote get repository. You do a couple of other things as well like we have a specific get ignore file just for what gets pushed upon pantheons get repo. So RCI will actually copy that file to be instead of like a pantheon and don't get ignore to don't get ignore. And that lets us prune out lots of things that we don't necessarily want to be pushed up that tests or secret things that probably shouldn't be checked in. And then there's say there's a few kind of quirks about the Pantheon platform that CI really helps us with. One of which is that Pantheon doesn't support environment variables.

Sally Young:

It does have something you can use with so terminus is it CLI tool that they give you for managing pantheon as a plugin without quote secrets and it essentially fakes environment variables by sticking them into your private files directory. You won't be able to use them as environment variables in your application. So you usually got to write like a little bit of glue code within Drupal to, to actually get those values out but it kind of works but a CI can really help you with that so you can have something that will loop through a certain set of environment variables that you set in your CI and then set them in your environment. So if you need to update those values you would just update them in your CI environment first and then run the bills. Okay. Through that way.

Sally Young:

It also can help us with the releases. So Pantheon has this like three Ted Dev stage live system so if you don't really want to use any of those environments in between which I have done on a couple of projects now because when you push something to master on the remote Pantheon Repo that goes to Dev. And then if you want to go any further you have to specific to create releases that one thing you can't do is kind of very easily bypass that and push something to production without it going through Dev first you can kind of create branches and add tags and sort of a [inaudible 00:50:40] but there's a sort of a few consequences to that.

Sally Young:

Like engine x doesn't restart propellor example. So and projects I've done where we want to have red sorry. Blue, green environments for example we're just kind of skip that whole Dev stage prod process and Devin stage do nothing and we just use the live environments in which case yeah, it's really helpful for us there because we can work it through all of those deployment steps without having to go into the Pantheon Dashboard and do them manually every time. And the terminus CI tool they provide is really, really useful for that. So you can install it in your CI environment and then interface with Pantheon really well.

Mike Herchel:

What is, you said blue, green environment.

Sally Young:

So if you have two production environments and in order to do a deployment you don't want to bring down your environment for any time at all. So you can do what's called blue green where if blue is live right now, we'll build on your site to the green stack and then when that's completely ready we'll switch the DNS over from blue to green. And so they use us will never notice any kind of downtime. What will probably happen is that the editors will have the downtime instead because they'll probably be some shuffling around of databases because we can't have editors editing in both places obviously.

Mike Herchel:

Okay.

Andrew Berry:

I hadn't heard that coming back into fashion. But it sounds a lot like changing, assembling on a patchy old school.

Sally Young:

Yes it is.

Andrew Berry:

It's more common in systems that don't have user data being edited all the time. And the editors really are just use who is editing your data. And most of the clients we end up working with [inaudible 00:52:36] for them ends up meaning [inaudible 00:52:38] for editors and not the general public because they're already serving out cash copies of the sites your varnish and Akamai and so on. So with a lot of Drupal builds like doing totally separate environments like that is effectively the same thing as just letting it serve from [inaudible 00:52:55]. But the nice thing is let's say you've got a deployment that takes 15 minutes because you have some big database change that you have to make and then it goes wrong and you need to roll back. You're not having to roll back in the same environment you've just never flipped the DNS in the first place. So there are some advantages that way.

Sally Young:

And [inaudible 00:53:16] also gives you a CDM by default like falsely. So if you're just doing little updates, again it's one of the things like James said, it's like don't feel like we have to do all of these things. If you're just doing smaller kind of updates then I think Pantheon will actually [inaudible 00:53:32] that for you at the CDN. So you just use as if they're not logged in and see any downtime.

Mike Herchel:

So what about maybe in a non Acquia Pantheon environment. Andrew if you were to maybe set one up and you're going to use, you're not, you don't want Pantheon Acquia or anything somewhere. What are you going to do?

Andrew Berry:

Yeah.

Mike Herchel:

How's it going to work?

Andrew Berry:

I mean, I guess I would say from a ... There's these two rules that I sort of follow when I am investigating these sorts of things for clients. And one is the hosting environment should be as simple as you can possibly make it. In my experience even some of the biggest Drupal sites out there would be perfectly well hosted by two to five [inaudible 00:54:24] instances with fastly or cloud flare or something in front of them. In comparison to like a really complex AWS, terraform, et Cetera, set up. And then on top of that is as much as possible everything should be in get.

Andrew Berry:

And that's really a prerequisite to true sort of CI nirvana in the sense of like, one, it allows you to test your changes before they go into production. So if you imagine you need to, if your host health hosting, you might need to update Apache or, or you know, updates are probably not as good an example, but maybe you're installing a new PHP module, like a Mongo db support or something like that. You want to be able to test that through CI and make sure that nothing breaks with me here hosting infrastructure. And if you're just doing that by hand by SSH into a server and installing the packages then you're not able to do that. So, I'm actually on a project right now where the client doesn't have any environment set up.

Andrew Berry:

They are going to be self hosting it internally. But we've been developing for I don't know, four months now and, there's just, there's no environments. And so what we do there is we're using circles CI to do all of the builds all of the tests. Right now it's running unit tests, it's running B tests and then another category it's called existing site tests, which is this wonderful Drupal library that lets you run against a basis. You write Drupal tests against a real database without having to go through Be hat. And we're using it this is a decoupled site so we're using that to validate API responses from Drupal. But then on the other side of it and again completely managed in get, we have a tugboat set up which was mentioned earlier to act as our hosting environment.

Andrew Berry:

So, for example we have a ticket open right now that's updating from PHP seven dot two to seven dot three. And so we open a pull request, we update the docker containers both for tugboat and circle to go to seven dot three instead of seven dot two and we let all the tests run and we click around the site to make sure nothing's horribly broken. And then we know it's good to go and you know, for, from my perspective if you're self hosting that should be your goal. You should be able to open a pull request and do something like update the PHP version and click around on your site, on your actual hosting environment and make sure everything works correctly. And I would definitely say that depends on the size and budget of your team.

Andrew Berry:

If you're really just a smaller site with one or two developers or even less than one developer like the amount of time it takes to get that level of automation is just not worth it. But if you're on a team of, five, 10, 15, 20 developers by the time you include front end, back end ops people project managers and all of that like having it all sort of in version control will really enable a proper sort of continuous integration workflow that doesn't have annoying manual steps that are frustrating and tricky to debug.

Mike Herchel:

Brings it up.

Sally Young:

So say that-

Mike Herchel:

I've interrupted you enough, Sally go ahead.

Sally Young:

I would also say that whilst it is nice to be able to host anywhere it can be tricky if he wants to have more than one web head just because triple expects this ephemeral file system to be there. And unless you want to be managing NFS, or cluster FS that can get really tricky. You can back it up with science you can have it on something like S three, but that has its own problems because Drupal assumes that the file system is local in many places and so it can act very slow because it thinks it's there and it's like Twig, cache files and all kinds of things.

Andrew Berry:

Yeah. And even something like, I've set up sites before on li node where, using NFS for a file sharing between say two web heads is pretty straightforward if you've done any sort of CIS admin work before. It's basically one line in your exports file and you can use the private IP address that li node provisions for you, it's really a case of what your expertise is. If you're a Drupal developer who's line of knowledge ends at once it's on the server* I'm done, then obviously like self hosting becomes way more daunting and it's definitely worth looking at some hosted service that's out there.

Andrew Berry:

But for a lot of other people, you may be are coming from a position where you're not just the developer, you are the in charge of all the website content. You're in charge of the hosting, you're in charge of backups and management. You know, it's one person is doing everything in which case it's really like I mean, honestly hosting Drupal these days is not any more complicated than it was 10 years ago. Just in a lot of cases we choose to make it more complicated for noticeably good reasons, but not always.

Sally Young:

And then again. It was like [inaudible 00:59:57] like yeah it's one line to set up NFS, but are you going to be confident when it goes wrong that you can recover from that? How fix it.

Mike Herchel:

Sally, you've worked on a number of your couple of Drupal projects. How does the CEI process differ from like maybe a traditional, you know, Drupal instance?

Sally Young:

So there's lots of different environments that you need to test, you have your triple stack only a PHP stuff and then there's probably a node stack that you need to test and so they're going to have completely different I mean if it was docker, it'd be completely different base images probably. And then they're probably also going to go to different hosts and they're going to have all their requirements and set ups and stuff. Because for example, if a guy independency and they don't notice any node things and so you might have to go to hierarchy for stuff like that. And also just the proliferation of everything being a different cloud service. The more microservices you have, the more that your CI is going to have to interact with. Whatever maintenance tasks needs to do for all of those. Other than that it's, you know, not too different.

Andrew Berry:

I think a lot of it depends on how your teams are structured. Like in my experience if you're building a decoupled site hopefully that's because you've got a ton of different consumers who are separate teams who are consuming a stable API and you know, you've got your IOS app team and your android app team in your web app team. In which case as a Drupal developer it might actually be, it might actually simplify your CI workflow because you don't care about anything. Beyond say the editorial experience and the VPIs are working correctly. It might not be your responsibility to tests that the website front end is working. Or likewise if you're working on the front end teams, maybe you have mock servers that are just providing static API data so you're not dependent on that [inaudible 01:02:02] in terms of Dev stage prod QA workflows and so on.

Andrew Berry:

I would definitely say like, if you are a team where you're doing everything, Sally's right on that it's more complicated just because you have more stuff to deal with And even if your not dealing with separate technology stack, I worked with a team where they were decoupled but the front end was actually a PHP application that was just using twig to then render things out. So it was like Drupal feeding another PHP app then was going out to the public.

Andrew Berry:

You still got another PHP app that you asked to host or manage? For the project I'm on right now we're actually running a copy of the react app in tugboat so that it's pointed to the Drupal site that's running against the same pull requests. So basically for every pull request you get a full copy of Drupal you get a full copy of the react app and if you want to test things out and you can, and it works pretty well. And you it's one of those things where it's like once it's set up you generally don't have to spend a lot of time tweaking it which is nice. But I would say from a time investment standpoint it was probably twice as much time to set it up as if it had been a full stacked or full website just because you have to set up Drupal and then you have to change all your gears over to setting up node.

Sally Young:

It has reminded me of a really fun problem we have right now with a decoupled stack, when we spin up a for a pull request, each of the separate environments so CI goes off and does that. We actually need to connect each separate service to the relevant separate service that's been brought up. So, like we have this graph QL content API and so it needs to know what multi Dev environment it lives in on Pantheon.

Sally Young:

And so he has changed those environment variables and point them that way every time we spend them up and so we just started using the service called DNS simple because this extra did problem where we're using. So we have multi-lingual and translation in the front end app which has reaction. Each one of those is a separate domain, so we need to set up a separate sub domains, so every single PR that we spin up so that we can test all of those, made sure it works. So we have our CI going to, do simple DNS I think. And it sort of goes off on every poll request and ties them all down and test all that stuff too which is just would be impossible to do if we didn't have it.

Mike Herchel:

So, Hey, who here wants to talk a little bit about tugboat? We mentioned a couple of times-

Matt Kleve:

Yes, tugboat, tugboat.qa. It's a Lullabot project.

Mike Herchel:

Product.

Matt Kleve:

Product. But that it's a Lullabot product. Deployment previews for every poll request is what the website says. So how does CI fit into a project that might use tugboat?

Mike Herchel:

Or how does tugboat fit into the prize? A project that might use CI?

Matt Kleve:

There you go.

Andrew Berry:

What's funny is tugboats started as this kind of brings things full circle. It started as a Jenkins install and a bunch of bash scripts. So there was a Jenkins plugin that was like a GitHub pull request builder. And we were working on a project for Intel at the time and the stakeholder for that project was very technical and could have set up the site and tested out all the work that we were doing. But he was also very busy and had like five of the projects that he was managing. And, so as I was staring at Jenkins I was thinking, what's this GitHub pull request builder thing? So I turned it on and I started just tinkering a little bit and created some glue code and Voila, like a couple of days later I had this very, very rudimentary like when someone would create a pull request it would build the Drupal site using recent database dumps and then comment back to get hub with a link to an environment where you could preview it.

Andrew Berry:

And that was the genesis of what later became tugboat and now it's, back then it was just PHP and Apache was in my SQL were all installed on the Jenkins box and now it's all Docker based and node based and a much more mature and I know almost nothing about how it works. However, the use case is still the same which is when work gets created by a developer and pushed up for review. And normally that's in the form of a pull request. Tugboat will detect that and create a preview of that work before and with it actually merged into it. Like, if you create a branch off of master and then you create a pull request to get that merged back into master tugboat, we'll take that branch merge it into master. So it is a true continuous integration because it's integrating the work with the work that may have changed, things that may have changed since the developers started that work. And then provide a way for QA for stakeholders to sign off on it by the providing a link to that work.

Mike Herchel:

And now it works with a providers like get lab in bit bucket and maybe your own get server right?

Andrew Berry:

Yeah, you can have at work with GitHub, GitLab, bit bucket or a private hosted Repo and you can even have it the enterprise. You can have it behind your VPN or behind your private network.

Mike Herchel:

And like to configure it's just like basically a big YAML configuration file where you specify the docker containers and like maybe the scripts that run within Docker containers. So you might have a Docker container for like PHP. And another for my SQL or something like that. Right?

Andrew Berry:

Yeah. It's similar to some of the other CI tools out there. And in that you're using a lot of YAML nastiness.

Mike Herchel:

It's a blessing and a curse to, YAML is a blessing and a curse.

Andrew Berry:

Yeah and then those YAML files are committed to the Repo so you can change it. And something that's relatively new that I'm actually kind of excited about with tugboat is that there's no a kind of really easy visual diff integrations. So like, you know, I make a CSS change but maybe the CSS code base is just like a pile of crap and I want to make sure that, I'm, maybe doing some refactoring but I don't want anything else to be affected. So what I can do is like within that YAML file I put in a section and I think it's like visual difFs or something. I specify a list of paths for it to defer to check. And what a tugboat will automatically do is it'll fire up, It'll look at the base preview which is like the current maybe version of master or something like that.

Andrew Berry:

And then it'll, it'll look at your particular branches of version and it will compare the paths for each of those. And then every single one of those paths for multiple breakpoints by default we have like three break points it will actually show images with a kind of the base preview and then your branch. And then it'll show a difference, and the difference is basically the changes highlighted in like some type of neon pink. So what I typically do is if I'm making changes like this, I basically look at this big long page, it'll link to a big page that has all the visual deaths and it takes a while for all the images to load. Then as you scroll down, you can look for some areas of pink that aren't supposed to be there and then when you look for these areas of pink, you can be like, oh no, it's creating... Why is it changing this margin? This has nothing to do with what I'm doing. And at that point you can go in and you can maybe fix that or make your selectors a little bit more specific or something like that. I think that's honestly pretty fantastic.

Andrew Berry:

The way I like to sort of break it down on the projects I'm on is that I'm talking about is for when you want something interactive, meaning I need a URL I can go and click around and do something with. And that stays until I basically decide that I'm done with it. Whereas most CI tools that are out there, you don't have direct access to the jobs that are running except in debugging scenarios. So you opened the polar quest they run the jobs, they report back and then they delete everything and clean everything up. Which is not, if you're building something which is a purely API code that might be all you need. But, as Drupal developers in the end we're building something which lives in a web browser that humans are going to use which means humans need to be in that loop somewhere.

Andrew Berry:

I actually on one recent project we actually are using CircleCI and tugboat together in a really neat way. So we have CircleCI configure it to run the content migration that every three hours. Where pulls down the latest database from the Drupal seven site runs the content migration with the latest coded Master and then it actually saves the resulting database into a Docker container. And we're using K.IO QUAY Is that how you pronounce that? I always get it mixed up-

Sally Young:

[inaudible 01:12:07] is key.

Andrew Berry:

Key. Yeah. Well there we go today I learned

Sally Young:

It looks like quay. That's like a marine thing.

Andrew Berry:

Yeah, it's like a what?

Sally Young:

Marina. Sorry. Oh my God!

Andrew Berry:

So it pushes the Docker container up with that database sort of committed inside of the docking container and then every time tugboat spins up it pulls down that Docker container with the migrated content already in it. So that way it only takes... It's very quick to, to spin up. And so having circled to sort of do all of the basically anything which is machine driven and then tugboat can use that to feed its own systems is... It works really well for us. And something else to dimension is that tugboat works with pretty much every type of technical stack, like including-

Sally Young:

Yeah.

Andrew Berry:

WordPress or-

Mike Herchel:

[inaudible 01:13:06] Drupal. Yes.

Andrew Berry:

[inaudible 01:13:07] Caravel.

Mike Herchel:

It has to be on Linux and Docker? Yeah.

Andrew Berry:

And that's like everything right? So maybe like, no. asp.net.

Mike Herchel:

Exactly. Well, you could use asp.net core on Linux now, so that'd be fine. Cool. All right.

Andrew Berry:

There are other tools out there too for like, I think tugboats niches is a large database backed whatever it might be. But there, if you're just doing like a spa, like a node application that doesn't really care about the data back end or whatever. Now.sh or maybe there's some other ones that Sally might know that.

Sally Young:

I was about to suggest now.sh.

Andrew Berry:

Yeah.

Sally Young:

They are amazing.

Andrew Berry:

There are other tools out there for things that aren't like a Drupal site or something that's, that's data backed. I think that's really where tugboat shines is something that has data that needs to also be a part of the review process. So it's not just reviewing the code but it's being integrated with other people's and with the data that's coming from prod.

Mike Herchel:

And again, for me at least, it's one of those things where it's like anything is better than a shared development environment [inaudible 01:14:22] Yeah.

Andrew Berry:

Yes, in my mind like as soon as you get to a world where you have environments per QA or say per pull request that you can click around and then that is automated, like that is just such a step ahead of manual or shared environments. There's nothing that I find more frustrating than joining a project and finding out that oh, it's going to take us an additional three weeks to spin up a second QA environment because we've got another team who's working on some totally unrelated part of the site but they already own this environment and we can't be stepping on each other's toes.

Mike Herchel:

So as we point toward wrapping things up I do want to just point out one thing. I think Sally mentioned something about it earlier but I just wanted to kind of make a good clarification. One thing that was said earlier when we talked about having this big bash or folder full of bash scripts is that you might have passwords and other secrets in there how do we handle that? In this era? Like what's the best way to store secrets that everybody needs to have but we don't need to be spreading secrets.

Sally Young:

Yeah. So first If you can avoid having them on your developers machines that's excellent. You should definitely do that So if you have a key for API access developers should have their own keys probably with a to a different environment or a restricted dataset kind of stuff. And for any credentials don't pass it around in slack, use a password manager and one password and last pass have sharing facilities for teams for that. And then when it comes to actually injecting them most CI tools will have some way you can put environment variables they won't show up in the UI and cycle CI for example, so you can pop them in there and then have it set it in whatever environment in these to. The problem with that iS, is there anyone who has access to USCI environment now has access to those secrets.

Sally Young:

So if you let your developers SSH into two containers, they can now get into those secrets. I think one way around that with CircleCI is that they have different contexts so you can protect certain environment variables. So like they wouldn't be available in certain jobs and things like that. So you could say I need this like production deploy context can they access this set of environment variables and you think most developers actually wouldn't need to SSH and to debug that one that's why you're doing it. And that's kind of, I'm coming at it from the context of the service I was pushing to not supporting environment variables properly for hierarchy for example, you can just set them with the rocky, and then you don't really need to mess around with them too much.

Andrew Berry:

I think in, when I think of it from sort of a best practices standpoint like, the not sharing secrets is sort of the step one that I think if you do that you are already doing fairly well. Meaning don't have simple, easy to guess passwords for Admin, access to API services and so on. Like make sure they're unique, make sure that if you're doling out password save for development environments if you have the ability to make them so they only have read only access to the API instead of rewrite. If you don't need right access then just don't grant that in the first place. It really helps reduce the attack surface because the other thing to consider is, and this is more if you're setting up a public project, but maybe you're working on a Drupal module or a PHP library or in Java script library and you're setting up continuous integration for it is what happens for polar requests from the rest of the world.

Andrew Berry:

And most tools have some way to control how secrets are handled. So, there's usually an option which can say, don't pass environment variables to pull requests from forks. Meaning if I just fork your repository and submit a pull request, it might run the jobs but then it might skip anything that's functional and integration tests that requires API keys. That matterS lessons sort of a private scenario where you implicitly are trusting all your developers because they're employees and coworkers and so on and there are other ways to get into those things. But it also depends on what kind of data you're handling.

Sally Young:

And bear in mind employees can leave as well. So if someone leaves under unfortunate circumstances, if you haven't rotated your keys and you've not given them a unique key then they still have access to all of those things.

Andrew Berry:

Which brings up another point that's, that's about best practice as far as connecting all these various services is a lot of times what we'll do on for example, Pantheon if we need a CI to be able to connect to pantheon is to create a user, a Bot user is what we call them to integrate with Pantheon and just provide the permissions that we need there so that you're not putting your own token for pantheon into say tugboat or CircleCI or something like that, that needs to integrate with terminus. That way you otherwise, like if I go into CircleCI and I put my terminus token in there, my personal one now that CircleCI job doesn't only just have access to my project, but it also has this particular project. But it also has access to all my other projects that I might have been granted to so then any other developer that has access to that can now get into any other project that I'm on If that makes.

Sally Young:

Or you leave and delete your token

Andrew Berry:

Yeah. Exactly.

Sally Young:

And then suddenly everything breaks.

Andrew Berry:

Yeah.

Mike Herchel:

Yes.

Sally Young:

The other thing about it is create the vault user with an email address that can be accessed by other people. So like if it's all very well if I create this Bot user but if I've used sally.young@lullabot to do it or my own personal one and I leave, I can still get in there mess things up.

Andrew Berry:

Yeah. Ideally it would be owned something that's owned by the, in our case, since we're working with our clients we want something that the client owns so that if they're not locked in to us either when they move on, it's not like, oh goodness! We've got to transfer the ownership of this over to them. It's something that they own and they've given us access to and then therefore they can manage it.

Mike Herchel:

Lest go ahead and wrap it up. So one final question to keep it kind of short and sweet here is if you have one particular problem within continuous integration or something that always bugs you and you could just kind of get itself for free what is it? And maybe we'll start with Andrew.

Andrew Berry:

One problem that bugs me.

Mike Herchel:

Yeah. Something that's just a pain in the pain in the ass every single time.

Andrew Berry:

Yeah I'm trying to think of something that is you know like the thing that is really ridiculous about all this in the end is like we are basically dealing with a web of services. Like if you imagine what happens when you fall in a pull request you're dependent on say CircleCI, you're probably dependent somehow to do that on AWS or whoever their hosting platform is, you're dependent on composer and whoever their hosting platform is, you're dependent on NPM and whoever they're hosting platforms is, you're probably dependent on drupal.org and all that being up. And that's just thinking about building, like who knows what other services your specific site is integrating with as far as pulling in data or APIs and all that. So if any one of those goes down you file a pull request and your job has failed and you're like looking at blogs and seeing, oh it failed because drupal.org is undergoing maintenance for some reason.

Andrew Berry:

Or there's an outage in a camp download the C tools module or whatever it is it's supposed to be doing. But as a developer that always sucks up five to 10 minutes of time just to figure out what that is. As far as having it being solved it is really nice that some services like CircleCI, their status page is not just for themselves. They have include on their status page the status of every dependent service that they are dependent on, as well as services that are commonly used like say GitHub. So CircleCI might be having an outage because GitHub is not sending web hooks properly, which has happened more than once in the past. And so They'll reflect that in their status's effecting their service. But it's still you have that initial like I swear everything passed why did my job fail? Which is never fun.

Mike Herchel:

Got you.

Andrew Berry:

Yeah. We stand on the shoulders of shoulders.

Mike Herchel:

I love it. James, any pains in your world?

James Sansbury:

Oh yeah. I'm struggling to think of something. I mean, I think in general the challenge is deciding how much magic to introduce because it does while it's really convenient and makes things easy when it goes wrong you're still face to face with okay, there's only a select few of people understand really what's going on here. And even if that's not true ultimately like someone could open it up and see oh! Here's, I see where this is happening there's still I think an intimidation factor there for a lot of people they think like that whoever's working on the on the CIS stuff is some sort of magician or something and like they understand it and it's like way over my head. But it's really not like once you get in there you're like oh! Okay, this is just a YAML file that executes these certain commands at a certain point and sure it uses its own particular syntax. But I guess just getting back to the choosing the level of convenience that you want that's right for the project it's challenging always because the magic when it unravels is challenging to kind of figure out where it's broken and how to fix it.

Andrew Berry:

And then it's not just figuring that out for yourself, but also figuring it out with your own stakeholders. I've worked with teams where there's when it comes down to it these stakeholders will just say, well I don't care that the tests are failing deploy it to production anyways. Because in their mind that's more important or they would rather the team work on features and getting test passing or CI fixed again. And so in those cases it's a huge opportunity, like is it just a huge sunk cost that you spent this time setting up testing CI and all that just to not actually follow it when it's trying to tell you something important. So, you have to have buy in from your nontechnical stakeholders on the team as well.

Mike Herchel:

Sally, anything that ... Has it been on your side?

Sally Young:

Yeah, it would be really nice if we didn't try to turn YAML into a programming language. [inaudible 01:26:29] every movie too that comes out. It's like well, now you can see if statements in YAML and [inaudible 01:26:35] and you can do more and more things I'm like, "Ugh, it's just great by script."

Mike Herchel:

Great.

Matt Kleve:

All right one thing Mike, that I did tease before the break, and I wanted to get to it was we talked about an ideal setup. Real quick for each of you talk about what an ideal setup would look like these days. Let's start with the Andrew.

Andrew Berry:

Yeah, I mean it's basically probably CircleCI, Tugboat, Robo tests for everything, and time allocated to actually write the tests.

Matt Kleve:

Sally?

Sally Young:

I [inaudible 01:27:13], sounds I got CircleCI and Docker I'm good.

Matt Kleve:

James?

James Sansbury:

I'd say GitHub, CircleCI, Tugboat, and I agree with Andrew having the time to write the tests that we need. Definitely.

Matt Kleve:

Mike, do you have anything that you'd like to add or maybe what it would look like for you?

Mike Herchel:

Yeah, I mean from my point of view, typically the back end developer on my project typically specify out. But like for me I just like to ensure that the CI process is compiling or transpiling my SAS and ESX style Java script just so I don't have to commit my CSS to the Repo and get tons of merge conflicts and stuff like that.

Matt Kleve:

My ideal setup involves one of these nice people doing it for me.

Mike Herchel:

Yes. I agree. If I could have Andrew or Sally be on a project with them and then I could just sit back.

Matt Kleve:

Thanks everybody.

Andrew Berry:

Thank you.

James Sansbury:

Thank you.

Mike Herchel:

Thanks everybody. Bye.