31 Days of Drupal Migrations with Mauricio Dinarte

0:00 0:00

Matt and Mike are joined by Mauricio Dinarte, who recently completed his "31 Days of Drupal Migrations' series, as well as migration-expert April Sides.

Episode Guests

Mauricio Dinarte

Thumbnail
Mauricio Dinarte is a passionate Drupal developer, consultant, and trainer with over 7 years of web development experience. After completing his BS in Computer Science, graduating with the highest GPA among 181 students, he completed a Masters in Business Administration.

April Sides

Thumbnail

April Sides is a seasoned Drupal Developer who is passionate about community building.

More about April

Mentioned in this Episode

Transcript

Transcript

Matt Kleve:
For September 27, 2019, it's the Lullabot Podcast! Hey everybody. It's the Lullabot Podcast episode 240. I'm Matt Kleve, a senior developer at Lullabot. With me is always co-host to the show senior front-end dev, Mike Herchel.
Hey Mike.
Mike Herchel:
Howdy, Matt.
Matt Kleve:
Hey. Mike, this is the Lullabot podcast. We talk about all things Lullabot. What is Lullabot? What do we talk about here? What do we do?
Mike Herchel:
That is a great question.
Matt Kleve:
What do you do here, Mike?
Mike Herchel:
Lullabot is a design, strategy and development firm that also now does support and maintenance if your company is looking for that [crosstalk 00:00:54].
Matt Kleve:
We should have that on the podcast at some point, right? We should talk to our support and maintenance team.
Mike Herchel:
That should be the commercial in the middle of this. We should talk to David Burns right after this maybe to plug that in.
Matt Kleve:
Okay. It's so much easier when I can steal Chris Albrecht's commercial. So yeah, but today we're not talking about that. Today, we are talking about something that I think really makes Drupal great. I think it's something that's vibrant, part of the Drupal ecosystem that's been around for a couple of generations of the software. It just keeps getting better and better. I think it's important for people using Drupal to be able to leverage it.
Mike Herchel:
Migration.
Matt Kleve:
Migration. Yeah, we're in that cycle of Drupal 8, where a lot... Drupal 8 has been around a while. The early adopters have already come and gone, and now it's the folks that are needing to bring their website into the current era, and migration from here to there is important, whether here is from Drupal 7 to the current version of Drupal 8, Drupal 6, Drupal whatever, or even some fancy other CMS. You can use Drupal's migration system that's built into core for the most part and bring your data into Drupal 8.
Mike Herchel:
Yeah, but do you know what's tough about migration is-
Matt Kleve:
Everything.
Mike Herchel:
... learning how to do it. With us today is a person who wrote this whole series of awesome articles on migration.
Matt Kleve:
Well, it's kind of documentation for migration.
Mike Herchel:
Yeah, yeah.
Matt Kleve:
Right. Well, it's documentation about his experience, what he knows, and sharing the knowledge.
Mike Herchel:
Mm-hmm (affirmative). So with us today, we have Mauricio Dinarte who has been involved with Drupal over nine years. He's the lead organizer of the Nicaraguan Drupal community, a Drupal developer at Agaric. Did I pronounce that right, Mauricio?
Mauricio Dinarte:
Correct.
Mike Herchel:
All right. Dinarcon on Twitter and Drupal.org. That's D-I-N-A-R-C-O-N. You wrote, what, a whole series called 31 Day of Migrations, right?
Mauricio Dinarte:
Yes. Hello, everyone. Correct. Throughout the month of August, I committed myself to write 31 articles on migration, one each day.
Mike Herchel:
Wow.
Mauricio Dinarte:
It was pretty fun.
Matt Kleve:
And you were also saying that it wasn't just in English, right, you're trying to share the knowledge.
Mauricio Dinarte:
Yeah. The series is part of a documentation project that my brother and I have started where we plan to write documentations about Drupal on different topics in English, in Spanish and French.
Matt Kleve:
Wow. Well, that's great. Also, with us today on the podcast, we have April Sides, a senior developer at Lullabot. She's here to kind of play the role of migration muscle. April has been recently on a project migrating dozens of individual Georgia government sites to Drupal 8. Hi April from Asheville, North Carolina.
April Sides:
Yeah. How is it going?
Mike Herchel:
April is also the organizer of the second best Drupal camp in the world, which is kind of mean.
Matt Kleve:
I didn't know she helped you, Mike, in Florida.
April Sides:
Burn.
Matt Kleve:
Glad you're here.
April Sides:
I'm glad to have me. That sounded really bad. I'm glad to be here.
Mike Herchel:
No, we're going to keep that first version in. Mauricio, let's talk about you first. How did you get involved with Drupal? I'm assuming since you are the lead organizer of Nicaraguan Drupal community you are from Nicaragua. How did you get involved with web development, Drupal. I remember seeing you many years ago at Florida Drupal Camp. You do a good amount of traveling. Can you tell us a little bit about yourself?
Mauricio Dinarte:
Sure. Yes, you're correct. I am from Nicaragua. I studied computer science at college, and it is incredible to me that during the five years program I never learned or received one development class. By my senior year, I was frustrated for not being able to create even a simple website. So, I started asking friends and looking for documentation. I remember buying a book on writing HTML and CSS, and that was my very efficient solution to writing websites. After that, some friends recommendations some CMSs. I tried a couple, but Drupal was the one that stuck. It was a [inaudible 00:05:39] site. I understood everything. Everything clicked at the time.
Mauricio Dinarte:
I feel that one of the reasons is that I found very good documentation. I remember using other CMSs trying to do simple things as what we call in Drupal Clean URLs. Maybe I didn't look hard enough, but I couldn't do it. With Drupal, it was right there. The documentation that was available, and I remember even back in the day listening to Lullabot podcast and all the resources. What got me into Drupal in the first place, being able to find documentation, being able to create somewhat complex websites without needing to write custom code at all because even though I knew some HTML and CSS back then, I wouldn't say I was able to do myself what I was able to accomplish with Drupal. That was the spark that got my interest. From them, I just kept learning and learning. At some point, I was able to write modules, to write things and here I am today.
Matt Kleve:
Mauricio, your 31 articles, you mention English, Spanish, and French. Spanish is your native language, right?
Mauricio Dinarte:
Yes. I speak Spanish natively.
Matt Kleve:
So, it makes sense that you would write an article in Spanish. So, English, you wanted to reach out people who spoke English. Explain French to me. Why French as well?
Mauricio Dinarte:
French is something that I have wanted to learn for years. I just thought that similar to anything, even the articles themselves on migrations, the best way to learn is by to teach or in this case to produce content in that language. At the start of this year, I started to take French lessons, not full-time but basically daily. As you can imagine, I just recently started learning the language, so I am not fluent enough to write the articles on my own. So, my brother, who actually speaks French, he is the one who makes the first translation, like the first pass and then I do editing and corrections. That's how we're able to write in three languages, but my personal goal is to being able to give sessions and trainings in French very soon.
Mike Herchel:
That's awesome.
Matt Kleve:
What's the motivation behind a series of articles about migrations? Why is that important to you and why is that something you wanted to target?
Mauricio Dinarte:
Well, to give a little bit of context, when I started my first migration project, I had been a Drupal developer, like being able to write modules, for example, for about five years. I remember it being kind of a frustrating experience. It was a very good Documental API and today it's one of the best Documental API from what I can see. But even though it was hard to understand all the different moving parts, I just, for one, want to save other people some time in trying to figure out the puzzle.
Mauricio Dinarte:
Also, I am very active in the migration selection channel, which is a good way to get support. I see a lot of common questions being asked over and over. I guess it's a common problem. It's a known problem that conversations get lost about a week after. So, I wanted to have a place to have a place where the documentation can be referenced permanently. If someone has a question, go read this article on the topic. It seems to be working because over the last month, I have been referencing people to the articles, and they have been finding them useful.
Matt Kleve:
April, you had a similar thing, right. Long time Drupal developer, all of a sudden we need to learn something about migrations. What was that like for you?
April Sides:
Great. So, the Georgia.gov migration was my first migration and that was just this past year.
Matt Kleve:
We're going to throw you to the wolves.
April Sides:
Yeah. I was really interested in it. I'd never done a migration. So yeah. The API documentation on Drupal.org is pretty good. There isn't a lot of documentation on the use of pseudo fields or temporary fields I found. I just did a presentation about some of the really challenging things that we did on the Georgia.gov project when I was at Drupal Camp Atlanta. And I just had to figure out like what is the actual of this technique of using a variable within the migration YAML file. I've seen it in multiple different ways, but I don't think any of it was on Drupal.org's actual documentation. So yeah. There's definitely some lacking information. This resource that you're creating is pretty cool.
Mauricio Dinarte:
Thank you.
Matt Kleve:
Mauricio, what was it like to write an article for 31 days in a row?
Mauricio Dinarte:
It was way harder than expected. This migration series is basically a brain dump of a migration training that we had given for a while. Lee and I thought that we will take a couple of hour each day to be able to produce the content, but I like to do research and to get deep on topics. So, sometimes to write one sentence on one paragraph, I had to do a lot of research. So, in the end-
Matt Kleve:
It sounds like the typical developer problem though. It's like, "Yeah, that will take a couple of hours." Then you dive into it, you're like, "No. To do this right, it's going to take a little more time than I kind of foresaw."
Mauricio Dinarte:
Yeah, for sure. On average, it took about eight hours to produce each article, some of them going up to 12. Those were the English version. I really liked it. I remember being enthusiastic each day when I was going to sit in front of the computer and start the new article, I start a next topic because even though I had something in mind, I would just spend a lot of time in the debugger checking how everything worked. Sometimes my assumptions were wrong. So, I was learning a lot throughout the series. Originally, it was supposed to be just whatever you know put it in text format, but in the end, it was a very good learning experience.
Mike Herchel:
I love situations like that. I find that I have similar situations when I write articles. Just the act of writing or teaching forces you to organize your thoughts, and it forces you to organize what you know. As you organize your stuff and put them in whatever order you're looking at, you find gaps. You find well, I don't know how this would work with this, or I don't know what works with this. That kind of forces you to open up a debugger or look at the documentation or do whatever. Just doing that is really powerful. I also will frequently reference my own articles and stuff like for code samples. Do you do that a lot?
Mauricio Dinarte:
Yeah, for sure.
Mike Herchel:
Yeah.
Mauricio Dinarte:
And as part of this migration series, I actually created a repository of about 28 samples that go along with each article, or most of them because some of them are theory. But that's something that I refer back myself, and it's something that someone who is learning they can just go and look at them.
Mike Herchel:
Yeah, definitely.
Matt Kleve:
We'll make sure we get some links in the show notes on the show node on Lullabot.com. Mauricio, is there something that you wish you had time to cover? Writing 31 articles seems like you should be able to get a lot done, but there's probably a lot that you ended up leaving out.
Mauricio Dinarte:
Yes. The focus of this initial series was not necessarily developers but site builders. So people who would be able to create complex migrations without having to write PHP code. That was a challenge trying to find the right balance between the deep technical content and what can be used by a site builder without having to write custom code. That being said, the series doesn't cover anything about writing custom plugins for migrations.
Mauricio Dinarte:
Another thing that we barely cover at the start in one article is the Drupal 6 to Drupal 7 upgrade process. As you said initially, the migration API in Drupal 4 can be used to migrate from any source. It can be previous version of Drupal like 6 or 7, or it can be node CSV files, JSON files, XML files, or the CMSs like WordPress and so many other use cases. The idea of the series is to give a general overview of how to migrate into Drupal, not focus on Drupal 6 and 7 to Drupal 8 upgrades. That by understanding the basic concepts as they're explained in the system, you'll be in good shape to start writing a specific upgrade process. That comes with its own challenges, so that's something that we were able to barely cover on a theoretical aspect, but we were not able to actually get into examples, and I would love to do that as well.
Mauricio Dinarte:
Another topic that's very asked is paragraph migrations. There was another article and several examples that work on top of that basic paragraph migration, but there are many other things that can be done with paragraph like nested paragraph, multilingual paragraph, paragraph revisions and things like that. That was not covered. The example is kind of basic in that sense. Commerce is another big topic. For example, I was love to write a series on migrating from either over cut in Drupal 6 or the Commerce Kickstart distribution in 7 to Drupal 8. Other topics include media entities migration, organic groups or even just a general topic when you have language migrations how you can process a lot of data or how to schedule migrations. There was a lot that I was not able to cover.
Matt Kleve:
When you were becoming a migration expert, what do you think the hardest thing to learn was just for your knowledge?
Mauricio Dinarte:
I guess understanding the ETL process which is actually the very first article. Knowing that for any given migration, you only have one source and you only have one destinations and [crosstalk 00:16:54]-
Mike Herchel:
You said ETL process.
Mauricio Dinarte:
Correct.
Matt Kleve:
Yeah, what does that mean?
Mauricio Dinarte:
ETL stand for extract, transform and load. It's the same paradigm used even outside of Drupal to move content from A to B. The migrate API in core it's that it's an ETL implementation. So, understanding the basic idea for that. As I said before, even though you have so many options to choose from for your source and for your destinations, for any given migration, you pick one and one. Then you do all the transformation needed to go from your source to your destination. So you can have multiple process plugins is what they're called. When I was able to understand that, everything started to click.
Mauricio Dinarte:
As I explained in one of the articles, my recommendation is to start with the very basics. Make sure you're able to read from your source. Make sure you're able to write to your destination and migrate one filler at time. For example, if you're doing an old migration, just migrate the title. Make sure that that works before trying to do everything at once because if something fails, sometimes it gets difficult to debug.
Mike Herchel:
Yeah. April, what about you? When you were starting to learn to migrate API, was there any part of that that was more difficult than others that you can recall?
April Sides:
Yeah. So I heard you mention nested paragraphs. Nested paragraphs were quite a challenge.
Mike Herchel:
Yo dawg, I heard you like paragraphs.
April Sides:
Pretty much. Yeah, nested paragraphs were challenging. We figured out a solution and is that a lot of people are trying to come up with a solution for that. But with the Georgia.gov migration, we migrated them as embedded or as markup in the body field. So, we didn't actually move them to a paragraph entity in Drupal 8. So, that might be a little bit different than how other people are solving it. I think in this project, I wrote my first recursive functions as well, so I could just dive down into the nested paragraphs and get to the content that I really wanted.
Mike Herchel:
What about easier concepts or easier things, Mauricio? Is there anything that you said like, "Hey, that makes sense." "Hey, this is the way it should be," or?
Mauricio Dinarte:
I think that-
Mike Herchel:
You can simply say no to this, and I'll strike it from the record.
Mauricio Dinarte:
I think that there is a lot of magic in the migrate API. Sometimes I'm just amazed on how it works. For example, if you want to create an old migration, you basically write two lines, process destinations and then specify entity node. I think that's good because there's a lot of things that are taken for granted, but it's magical in the sense that you don't know what's happening. When things do not work out of the books, you might get complicated to understand on debug problems. Yes, it is very easy to write to different destinations, but when the resource are not what do you need or if something happens, it can get very complicated to debug. So, it's twofold.
Mike Herchel:
Hey, going back to the articles, one quick question, are these meant to be read and undo in order or can you just pick one up that you're looking for and read that?
Mauricio Dinarte:
The idea is that if you are completely new to migrations, you should read them in order to understand them because some of them deal with concepts that had already been explained. But if you're very familiar with migration maybe from D7 and you just want to learn the D8 way to do it because it changed a lot in between versions, you can go directly to the specific article covering the topic.
Mike Herchel:
Okay.
April Sides:
Question, so how much planning did you put into what articles you were going to write ahead of time, or did you just organically flow from the different articles that you wrote?
Mauricio Dinarte:
I guess, well for one, this is based out of a migration training that I have presented in the past. So, the flow is mostly the same that I follow in the training. That being said, these articles cover way more content. In the training, we don't cover so many sources, for example, that are explained in the article. Like probably a month ahead, I started planning on what topics I wanted to cover and in what order, having the idea of these will help me explain that, and these concepts are fundamentals to understand what comes next.
Mauricio Dinarte:
I was intentional in the order that I presented things. You can group them if you see them in order, like understanding basic concepts, then understanding how to migrate into different destinations, how to migrate from different sources, how to manage configuration with Migrate Plus, how to develop migration. They're also bundling in pieces, but you can read them in any order if you're already familiar with the API.
Mike Herchel:
We're talking migrations on the Lullabot podcast. Coming up right after this, we're going to get a little deeper into the technical aspects of making a migration happen, things that April and Mauricio like to see when they're working on migrations. Also, since I think we have three community organizers together, I think we'd be reminisced if we didn't talk a little bit about your local camps coming up right after this.
Speaker 5: Whether you're learning how to build sites with Drupal or diving into the code, there are community powered camps, summits, springs and trainings happening all over the world. Find all of these and more at drupical.com. Of course, if you want to boost your Drupal charts from the comfort of your own home, point your browser to Drupalize.Me and stuff your brain full of carefully crafted videos and tutorials.
Mike Herchel:
Welcome back to the Lullabot podcast. We're talking a lot of things including migrations. We're talking about 31 Days of Migrations, writing 31 articles over 31 days, which to me seems very crazy. We might be talking a little bit of a community, but first let's go deep into the technical aspects of migrations. What exactly is migration? How do we start doing that in Drupal? I'm guessing we enabled some core modules and what exists in Contrib. April, why don't you lead it off since we've been talking with Mauricio for a good amount?
April Sides:
Yeah. For the Georgia.gov site, we used the migrate Drupal modules and core. Then there's, let's see, one, two, three, four, five, like five modules in the Contrib space that we used. One was migrate file that gave us some extended capabilities for migrating files. Migrate Plus is very key and is pretty well documented along with the core modules. There's Migrate Source CSV because we had some CSV data for some default values, Migrate Tools, and the Drupal upgrade module as well.
Mike Herchel:
So, you turn on these modules. Where do you get started? What do you do?
Matt Kleve:
They just work TM.
Mike Herchel:
They just work?
Matt Kleve:
Yeah.
Mike Herchel:
Well, that is pretty magic.
Matt Kleve:
No, I don't know.
Mauricio Dinarte:
One important thing about understanding the ecosystem is that what each module provides the core migrate module provides the basic API for migrating data. This is not a specific to a particular source or destinations. Then you have migrate Drupal, which is especially if we're clear about upgrading from Drupal 6 to Drupal 7. It also has some support for Drupal 8 configuration entities to be moved from one site to another. Then you have the option to choose how to want to execute your migration. In core, you also have Drupal migrate UI, which provides a user interface where you can put your credentials to Drupal 7 site, and it will read the configuration from any active modules from both ends and try to migrate all the data and configuration as possible.
Mauricio Dinarte:
If you don't want to do that or if you want to have more control, April just mentioned migrate a big module, this gives command line support. For example, using DOS, you're able to provide credentials again to the Drupal 7, and you can both two things. One just like with Drupal migrate UI, you can instruct it to face all the configuration and all the content and executive everything. Or if it's often the case, you need to change the architecture of your site. You can instruct the module to give you the migration definition files and before the configuration files only, and you can modify those files to accommodate for any change in field names or field types or anything that might needed in the process and then you execute your changes.
Mauricio Dinarte:
I actually wrote an article about Drupal modules that support migrations. I remember this is one of those where I spent a lot of time, way more than expected, because there are like about 600 modules and basically review and test it around 200 of them. Not all of them made it into the list, but I found a lot of things that I didn't know about in the process. One of them is being able to run migrations in different ways. So you can also run migrations on a schedule. For example, if you need something to happen every day, like an RSS feed that you're reading, you can use migrate scheduler or Migrate Cron for that.
Mauricio Dinarte:
If you're doing upgrades, there is some module called migrate manifest that will ask to list a lot of migrations that you want to execute and group them together. You have Migrate Tools and Migrate Run that gives you the command line helpers. Migrate Tool itself gives you a UI to execute migration from the UI when they're defined through configuration. That's an important part that we might cover later. There are automotives like Microsoft UI that allows for specific types of migrations to do them through the UI. Like to download a file and execute the migration through the UI.
Matt Kleve:
Mauricio, tell me you're reading this off of a list.
Mauricio Dinarte:
I am.
Matt Kleve:
Okay.
Mauricio Dinarte:
I have my article open.
Matt Kleve:
Okay because if you were reciting that from memory, I was going to be quite impressed.
Mauricio Dinarte:
Not quite.
Matt Kleve:
Which article do you have there that's talking about?
Mauricio Dinarte:
It's actually called List of Migration Related Drupal Modules.
Matt Kleve:
Great. That-
Mike Herchel:
Yeah, that's very good for SEO.
Matt Kleve:
I was a little concerned that you had that all in memory, and you knew the name of every Contrib and what they did. I was going to be really impressed.
Mauricio Dinarte:
No, there are more like 600.
Matt Kleve:
Just like Drupal Contrib, some of them work better than others, and some of them are documented better than others, and your mileage may vary, right.
Mauricio Dinarte:
Yes, for sure. That's why there are at least 200 that are compatible with the Drupal API, some of them are in either very early stages or lack documentation completely.
Matt Kleve:
Sure.
Mauricio Dinarte:
[crosstalk 00:29:39].
Matt Kleve:
Yeah. I mean, everybody wants to try and give back, but they probably get as far as they need for the project they're working. They just committed to the ether and then, "Hey look, there's another module that could be used for migrating Drupal sites." So, it sounds like there's a lot of flexibility involved, and it's not necessarily just, "Hey, upgrade your Drupal 7 or 6 site or bring your WordPress site over." You could migrate whatever you want. It's kind of whatever you need. April, you've been doing stuff, migrating content in from APIs and authentication and lots of little tricky things. How does that work?
April Sides:
Yeah. With the current project I'm working on, we have various API endpoints that we're going to be migrating content from. Some of them have things like basic authentication on top of it. So, there is built in within the system that you can identify your credentials so that the Drupal migration system has access to that endpoint data because you're not going to manually be doing it, the system itself has to be able to access the data. That was pretty cool trying to figure out those complexities of accessing data, and just looking at different data instead of just looking at database driven data. Looking more at JSON formatted data was pretty cool.
April Sides:
We actually did use the Migrate Source UI module so that the client, on Georgia.gov, could import things after the actual migration of the site happened, so they could continue to use that sort of feature. That was a pretty lightweight module that you can provide migrations for. I like that one.
Matt Kleve:
You kind of alluded to running the migration. Can we talk a little bit of how that actually works these days? For a while, it was Drush command. Part of that as I recall you could some from the UI, how do you run migrations these days? How does it work?
Mauricio Dinarte:
So, you're still able to it either from the user interface or through the command line. The one caveat is that you need to understand if your migration was defined as configuration or not.
Matt Kleve:
What does that mean? Explain that to me if your migration was configuration?
Mauricio Dinarte:
So, your regular migrations as YAML files, and the YAML files can be placed in two locations. It can be /migrations directory in your module or in a /config/install. When you put in /config-
Matt Kleve:
It's slash what, config what?
Mauricio Dinarte:
Config/install.
Matt Kleve:
So, it's the config directory where all the YAML ends up. You export config to-
Mauricio Dinarte:
Correct.
Matt Kleve:
Or it can be inside your custom module in a migration directory.
Mauricio Dinarte:
Actually, in both cases, you would have, inside your custom modules, either the /migrations directory or inside your custom module /config/install. I guess I'm not explaining this well. Let me roll back a little bit.
Matt Kleve:
Maybe you should write a 30 second article.
Mauricio Dinarte:
Yap. Actually, let me open the articles. Okay. I think I-
Matt Kleve:
So, you're actually using it to reference what you want to talk about now.
Mauricio Dinarte:
Exactly.
Matt Kleve:
It sounds useful.
Mauricio Dinarte:
There are two ways to run migrations. One is through the command line or through the UI. If you want to run them through the UI, they need to be migration in the finest configuration. These are provided with the Migrate Plus modules. These modules gives you the opportunity to the same migration definition migration, to put them in config/install as it is any other configuration managed through the CMI system in Drupal. When you do that, you have a couple of more options. One of the options that, because it is just configuration, you're able to use any module in the ecosystem of configuration management to overwrite paragraph migration.
Mauricio Dinarte:
For example, April was talking about authentication. You were very lucky not going to want to add your credentials in the migration definition file because those will go to your repository, and you don't want that plain text. One option when you have your migration definition configuration is that through custom code, you can read configuration environment, for example, and update the configuration, like inject code into the migration before they are ran and executed. So, you inject the credentials who having to put them in plain text, for example. This is something that is provided by Migrate Plus. When you have this module, you can manage your migrations as configurations. Again, you can use any module like config. Yeah. You can use any module in the CMI ecosystem to manage the migrations.
Matt Kleve:
Okay.
Mauricio Dinarte:
Going back to the original questions about how to execute them, when you have them as configuration, the Migrate Tools module, we read those, expose a user interface where you can execute them like import, roll back, and so on. Then you have Drush commands provided by either by Migrate Tools or Migrate Run to execute from the command line no matter the type of migration it is.
Matt Kleve:
It sounds to me like if I wanted to be in a browser, it was probably because I was developing or testing migrations. But often when you're running migrations it's happening as part of an automated task, so running it from Drush makes a lot of sense.
Mauricio Dinarte:
Yes. There's definitely a lot more flexibility when you execute migrations from the command line, but that's one of the big selling points. If we go back to my original story, that's what got me into Drupal, being able to do a lot without having to code or without having to understand the command line. When we give the training, we are basically able to migrate a full site without ever touching the command line if you don't want to. So, that's a very powerful tool to give to site builders.
Matt Kleve:
So, you know command line training, does it involve writing custom code?
Mauricio Dinarte:
No.
Matt Kleve:
Wow.
Mauricio Dinarte:
It's all through the UI. We also code a little bit depending on the length of the training. If it is like a half day, it's only UI. If it is full day, we also cover command line stuff and [crosstalk 00:36:33].
Matt Kleve:
You also just migrating some basic fields from here to there, right? You're not doing complex data manipulation or anything like that.
Mauricio Dinarte:
Well the-
Matt Kleve:
I guess it depends on the plugins that you're adding from the Contrib space, right?
Mauricio Dinarte:
Yes. Yes. The repository for the training is open source. So we can also link in from the show notes. Yeah. We do basic paragraph migration, addresses, basic field mappings. It's basically the training is similarly to the migration, like trying to explain the concepts, so you understand the basic logic of how it works, and then you can build on your own.
Matt Kleve:
Do I need to come to Nicaragua for your training?
Mauricio Dinarte:
Surely not, but if you can yeah. Here in Nicaragua, the community is not as developed, so we usually give start level trainings. We rarely cover migration and specific tops, but even that being said, even in those basic trainings, we use the migrate API to bring in default content that can be used to explain other things like [inaudible 00:37:53], for example.
Matt Kleve:
Okay.
Mauricio Dinarte:
But you can attend camps where we usually give the training.
Matt Kleve:
So, when it all goes wrong, and you have a migration working, and you're having problems, what do you do to debug a migration? I want to hear from you both but let's start with April.
April Sides:
Xdebug is your friend.
Matt Kleve:
So you're talking just because it's all running in PHP. You're not necessarily writing a procedural function to migrate from here to there. All of the code is there, and you're saying Xdebug, right? Just step through line by line.
April Sides:
Right. Within your custom module, you can have, in your dot module file, there are a few hook related or hooks that you can use to hook into the process. You can drop in break points there and see what your source roll looks like to see what the data looks like. You can manipulate it from there too if you need to. Then there are other, you can add break points directly into the migrate modules to see what's happening there. Yeah, Xdebug is your friend.
Matt Kleve:
So, it's like, "How do you debug if you ever have a Drupal problem?" "Xdebug." Mike says, "No, I use the console in the browser."
Mike Herchel:
I am way cooler than that.
Matt Kleve:
Mauricio, do you have any other thoughts about debugging migration strangeness?
Mauricio Dinarte:
I agree that proper debugger is your best friend. One thing that I tend to do before going into the debugger is trying to analyze the problem from a general sense. I wrote three because they're related to debugging migrations. For example, one of the things that I recommend is minimizing the surface for errors. That's tied back to what I said at the beginning; try to get the minimum migration working and build on top of that.
Mauricio Dinarte:
Sometimes when you have 10 fields at a time, single migration on one of them fails. If you have 10 options to look for than if you only need to check one. Another thing that was one of those confusing things for me at the beginning is, which module do I need for X or for Y. So, there are a lot of plugins that are only available when you have the specific modules. When you have a reference from an example online, and you try to copy and use that other thing, it might be using plugins that you don't have available locally. So, making sure that you have all the modules required for the task is also useful.
Mauricio Dinarte:
Another thing is again checking if you are managing your migrations as good on configuration because just placing it in the wrong place might give unexpected results. Another thing is that you should read the official documentation. Sometimes there are some options when you are configuring a plugin that are not required, but they're super useful for different use cases. So, understanding your tools and making sure that you know all the possibilities is always a good idea.
Mauricio Dinarte:
This might be a little bit more involved, but try to be on top of how the ecosystem changes. For example, recently there were new releases for Migrate Source CSV and Migrate Plus. In those new releases, there are were some breaking changes. Again, if you're referencing an online article, and it is using another version, things might break. Also, reaching out for help in Drupal Slack, pound migrate, it's incredible the amount of support that you get from migrate maintenance and just people working with the API in general. So, even though yes, and I highly recommend and that's how basically the tool that I use to write the series, use a proper debugger.
Mauricio Dinarte:
There are other things that you can do before actually diving into the code. When you have to dive into the code, again there are process plugins that can help you print the information as it is being processed. Then the best thing to do is configuring a proper debugger, which there is also an article for that in the series.
Matt Kleve:
It sounds like these migrations can be as complex as you want them to be, April. So you can group migrations? Explain that to me. How does that work?
April Sides:
Yeah. On Georgia.gov, we created a migration group where we could define for each individual YAML file can have dependency. Let's say a site page migration depends on a field related links migration that happened previously. So you can define those dependencies. Then for the entire site, each of these migrations has defined that it's a part of a certain group. So, you can run the group instead of running each individual migration, and it will just run through each on in order that it seems fitting based on the dependencies that you've set. Then you can roll back that group as well. That was one of the things we prioritized with the Georgia.gov migration because it was a multi-site migration. The ability to run the migration and roll it back pretty seamlessly was really important to our process.
Mauricio Dinarte:
Another thing that migration groups give us is the possibility to define shared configuration. For example, if you are using an endpoint, and assume endpoint or node JSON feed, for example, is going to be used for all your migrations. You can define that in one place, the migration group. That anything that you define there is going to be copied over to each individual migration before they are executed, so you don't have to repeat the same configuration over and over. If you ever need to make a change, to make the change in one place.
Mauricio Dinarte:
One thing to notice that migration groups are provided by the Migrate Plus module. So, if you want to use them, you need to enable this module. Another alternative that is provided in core is called migration tag, which is a similar idea. You had a tag for migration and then you can execute multiple migrations by tag. Tag, contrary to groups do not offer the option to have shared configuration, but they work out of the box without needing extra modules.
Matt Kleve:
Okay.
April Sides:
We're using tags I think in this current project to inject our authentication credentials. We look at the migrations to say, which migration is this? Does it need this basic authentication? We can get the environment variables and inject them so that those things are not being put into the code base.
Matt Kleve:
So, if the migration has a tag, you need to go seek out the authentication stuff?
April Sides:
Right.
Matt Kleve:
Okay.
April Sides:
It's some custom code that says, "This migration had this tag." Then we get those credentials and drop them into... Basically, we're adding and altering the migration configuration on the fly before it runs.
Matt Kleve:
The credentials are just admin and password though, right.
April Sides:
Yeah, but you don't want that in GitHub. It's a security mess, security.
Mike Herchel:
You don't want people to guess it yeah. Yeah.
Matt Kleve:
Or is it admin-admin? One or the other.
Mike Herchel:
All right. Well, let's wrap it up.
Matt Kleve:
So Mike, Florida Drupal Camp, is it best?
Mike Herchel:
Yeah, Florida Drupal camp is the best.
Matt Kleve:
Mauricio.
Mike Herchel:
Mauricio can validate that, correct?
Mauricio Dinarte:
I am actually using my Florida Drupal Camp t-shirt at the moment.
Matt Kleve:
Nice.
Mike Herchel:
Oh, look at that.
Matt Kleve:
Mauricio you're also involved in organizing community activities in Nicaragua, where you are.
Mauricio Dinarte:
Yes, correct.
Matt Kleve:
What is the city where you are?
Mauricio Dinarte:
Managua, the capital.
Matt Kleve:
Okay. What kind of community do you have there?
Mauricio Dinarte:
Well, we have been trying to grow the Drupal community for several years now. For many different reasons, it goes up and down from time to time, but just yesterday, we have a meetup with 20+ people. The interest in Drupal in general has been growing a lot lately. The last camp was organized two years ago, and we're planning to have either a Drupal camp or a small event at the end of the year. We've done only two events in the capital. We also do events in León, where actually one of the migrate maintainer lives [Luka Feri 00:47:16].
Mauricio Dinarte:
We also go to other series like basically we do a tour around different cities in the country trying to teach Drupal and general web development. Next month, we're going to participate in a program in a city called Granada, where they teach small girls to code. So, they're a program where they learn basic HTML, CSS, JavaScript, and they build it on websites for their communities. Now we're going to teach them how to use Drupal to be able to do more.
Mike Herchel:
That is awesome.
April Sides:
Yeah.
Matt Kleve:
April though has the other best Drupal camp or something?
Mike Herchel:
Second best.
Matt Kleve:
Second best. Yeah.
April Sides:
Yeah. Our camp in Asheville, North Carolina, I think we're the only camp in the state of North Carolina, but we're kind of a destination camp. We have a lot of people that come from outside of Asheville for the camp.
Matt Kleve:
That's because Asheville is Asheville. So yeah.
Mike Herchel:
Yeah, it's gorgeous.
April Sides:
Yeah. We have a lot of fun, really learns a lot, gets to share their knowledge and make friends and nice networking contexts. We also have a lot of social events.
Matt Kleve:
April, I saw you-
April Sides:
We have a hike on Sunday. So, it's always fun.
Matt Kleve:
I saw you raise your eyebrow when Mauricio said he had 20 people at a meetup.
April Sides:
Yes, a lot of people.
Matt Kleve:
Do you get to that?
April Sides:
No. Our local community is just not as engaged. We're having some difficult doing that. We're doing some just a monthly social meetup right now, and it's not really working either, but I think though that's great that you have 20 people. That's really awesome.
Matt Kleve:
Yeah, that's impressive. Mike, do get 20 people together in Gainesville on a regular basis?
Mike Herchel:
Yeah, but I run the front-end developers meetup here. It's kind of more generic.
Matt Kleve:
So, it's beyond Drupal, but you get a similar crowd.
Mike Herchel:
Yeah, Gainesville is a little smaller than either of the cities, but yeah.
Matt Kleve:
Nice.
Mike Herchel:
So you're saying that's even more impressive then?
Matt Kleve:
You know, I think it's less impressive because it's not a Drupal meetup.
Mike Herchel:
Yeah, it's more generic. So, we have people coming from various CMSs. We even have backend people trying to learn JavaScript and styling and things like that. So yeah. It's almost easier to attract people. Cool.
Matt Kleve:
Pretty good. Is there anything you'd like to add before wrapping this up Mauricio? What's your next series of articles going to be about? You're going to do another month or something?
Mauricio Dinarte:
I am currently working on another series.
Matt Kleve:
You're kidding. I was joking.
Mauricio Dinarte:
No, no, but I am not planning at least in the very near future to do another 31 days. That was too much to be honest. The current series it actually started before the migration one, and it's something that I have been wanting to do for years. It is about basic Drupal concepts. So, for a couple of years now, we have been part of the global training, the initiative, which is teaching the very basics of Drupal and building a website in a day if you will. I am currently working on that. I'm kind of one theory into the series, and I plan to finish it in the next month or so.
Mauricio Dinarte:
Another thing is that so far, we have only produced text, but we have plans on the understanddrupal.com website to produce eBooks and video tutorials. So, both for the migration series and for the understand Drupal basics. Once the articles are finished and translated in the three languages, we plan to create them as eBooks and video tutorials as well because different people learn in different ways. So, giving the option is something that we want to do, not only in terms of languages, but in terms of media.
Mauricio Dinarte:
That being said, I'd like to say thanks to Agaric, Drupalize.Me, and Centarro who were the sponsors of the migration series. Yeah. If you would like to sponsor any other series, get in touch.
Matt Kleve:
You said what was the first company?
Mauricio Dinarte:
Agaric.
Matt Kleve:
That's the company you work for.
Mauricio Dinarte:
Correct.
Matt Kleve:
And you mentioned Drupalize.Me-
Mauricio Dinarte:
Yes.
Matt Kleve:
... which are Lullabot friends.
Mauricio Dinarte:
Correct, and Centarro, which-
Matt Kleve:
Which is Ryan Szrama, right? The-
Mauricio Dinarte:
Right, Drupal Commerce.
Matt Kleve:
The Drupal Commerce folks. Cool. That's awesome. We're glad it's there. It's a good addition to the Drupal ecosystem.
April Sides:
Yeah, thank you.
Matt Kleve:
Yeah, thank you.
Mauricio Dinarte:
Thank you. Yeah. Eventually, I want to bring some, if not all, of the content into the official documentation because here it's nice to be able to have one place to reference everything.
Matt Kleve:
Yeah.
Mauricio Dinarte:
Definitely.
Matt Kleve:
April, you've been exploding your brain with migration knowledge for, I don't know, like a year now straight or so.
April Sides:
Yeah, it's so long.
Matt Kleve:
Are you looking forward to do something else, or you're okay in that world?
April Sides:
I really enjoy migrations. I think I've been converted into a data nerd. I like to move data around, and I like to figure things out. It's pretty cool.
Mike Herchel:
That's awesome.
Matt Kleve:
That's great. We need folks like you on projects.
Mauricio Dinarte:
Migrations are super fun, challenging in so many ways that it's hard to get bored. Every prog is different.
Matt Kleve:
Yeah, there's always a problem. Mike, are you ready to become a data nerd?
Mike Herchel:
Sure.
Matt Kleve:
Okay.
Mike Herchel:
Yeah. I mean, I like doing stuff. It's-
Matt Kleve:
Well, I think you should start day one on Mauricio's blog posts and start working too.
Mike Herchel:
I will add that to the list of articles and tutorials that I currently have on my list.
Matt Kleve:
Great.
Mike Herchel:
So, hopefully soon.
Matt Kleve:
Thank you for joining us.
Mike Herchel:
Thank you, Mauricio.
Mauricio Dinarte:
Thank you very much everyone.
Mike Herchel:
Thank you, April.
April Sides:
Thank you. Yeah.

Published in:

More Resources

See all

Getting Excited for Drupal 8 - The Donkey, Human, Ant, Chameleon CMS

Listen

The Imaginary Band of Drupal Rock Stars at Lullabot

Listen

Angie Byron on Drupal core and Spark

Listen