In the last version we added a new table country and enforced a one to many relationship between country and artist such that every artist belonged to a single country.
These types of database changes,
when you add a table and some columns,
tend to be some the easier types of migrations to perform.

Lets take on a more complicated migration problem,
and add some more fixture configurations and seed data.
This time we will add a table and change the nature of an existing relationship.
Is DBIx::Class::Migration up to the job?
Let's find out!

In version 1 and 2 the relationship between Artist and CD is one to many; for each Artist there are zero to many CDs,
and for each Cd there is one and only one artist.
Now we realize that a CD could have a bunch of Artists working together as a collaboration.
We need to change our database to reflect this need,
and we additionally need to migrate our existing data to this new schema.
Let's get to it!

So if you are familiar with DBIx::Class you'll already know this is a normal pattern. We have a bridge table where each row points to one Artist and on CD row, and no duplications are allowed (the Same Artist can't be linked to the same CD twice, for example).

Now we need to change the Artist and CD Result classes. Open your editor on lib/MusicBase/Schema/Result/Artist.pm:

So we didn't change the columns, but we removed the direct relationship to Cd and replaced it with a relationship to the new ArtistCd result class. We also created one pseudo 'many to many' relationship across the bridge so that we can directly pick up all the Cds for a given Artist.

Now we need to change the Cd. Open lib/MusicBase/Schema/Result/Cd.pm and make the following change:

Changes here are a bit deeper. First we removed the artist_fk column since we no longer constrain each Cd to a single Artist. Then we removed the relationship directly to Artist (since we no longer have that) and replaced it with a new relationship artist_cd_rs which connects us to the ArtistCD bridge table. Last, we added another pseudo relationship so that you can easily cross the bridge table and get all the Artists related to a given Cd.

If any of this is confusing to you, you should stop now and review the DBIx::Class manual, covering relationships.

As always, you should remember to up the $VERSION in your MusicBase::Schema:

So first off we've check the status of our system and can see the database is one version behind. Let's prepare the migration:

dbic-migration -Ilib prepare

And as usual we can see a bunch of new version 3 directories and some 2-3 directories related to upgrades. As in the previous migration, we'll skip the 3-2 downgrade, instead covering downgrades in a later section. Here's some of the new directory structure:

At this point there should be no surprises. You'll notice that prepare made a fresh all_tables.json for you, which has added your new bridge table and it also copied over your custom countries.json from the previous version. Please note that this is a dumb copy; no attempt has been made to convert the configuration to make sense with any database structural changes. You'll need to examine your custom configurations and manually make any needed changes. In this case the existing custom configuration is fine for this version, so we can move on to reviewing the deploy and upgrades.

There should be no surprises under deploy/3/*. The only reason you'll need to poke into these files is if you need to see the full DDL as part of helping you craft the upgrades correctly, or if your DBA needs to tweak aspects of the design (change indexes, storage engines, etc.)

The real fun begins under share/migrations/SQLite/upgrade/2-3/*. Again you have the 001-auto.sql file, which is the first, best guess on how to upgrade the database. We need to poke into that and shape it into something that can work for both our database structure and our data. Let's look at the the suggestion. Open share/migrations/SQLite/upgrade/2-3/001-auto.sql in your text editor:

This doesn't look too bad. From first review it looks to me like we are just missing transfering data from the old relationship to the new bridge table. So like last time we will bust this up into a few steps, under new file names but under this directory. Lets handle this in bits.

touch share/migrations/SQLite/upgrade/2-3/001-create_artist_cd.sql

And then open that in your editor. We will copy the first bit of the 001-auto.sql, having to do with creating the new table and indexes over:

Luckily everything we need exists in the current cd table, so this is a straightup insert. In this case I didn't use a Perl deploy run script since I felt the performance benefit of a native SQL approach outweighed the advantage of database portability. If I ever needed to make this work on say MySQL or Pg, I'd need to rewrite it, and there's not a lot of SQL so I am willing to take that risk.

Lastly, we need to alter the cd table to get rid of the now unneeded relationship:

touch share/migrations/SQLite/upgrade/2-3/003-alter_cd.sql

Then open that in your text editor, and lets bring over the last part from 001-auto.sql:

Again, since SQLite doesn't have any DDL alter to remove FK's we need to roundtrip the data via a temporary table. If you had a million+ rows you might worry about this approach :) Again, I will accept this DDL change for the scope of our limited requirement.

So, that's all the changes. Like last time, remember to remove the suggested upgrade script:

rm share/migrations/SQLite/upgrade/2-3/001-auto.sql

And you'd probably wish to commit these files now if you are using an source control system (and if you are not, prepare for pain!)

As you might notice above, even though the schema is version 2, we installed fixtures from version one. The tool will always try to match fixture populates to the current database version. And remember, if you don't tell populate which fixture set to restore, it will always use the all_tables set.

Just about when you are finished there is a sudden requirement change to the version. Or maybe you realize your new schema isn't exactly what you need. In any case you find yourself in a situation where you've already updated and now you need to step back, change the migration, and upgrade again.

This really isn't a problem at all. Its actually very easy to step back and redo your version. You could handle this in two ways. Either you've been very good and making sure you made good downgrades (we didn't :) ) or you can force install the database to an arbitrary older version and start again with the prepare command. We will take this second option for this tutorial.

So for the purposes of our tutorial, lets say that suddenly we realize our design for the country table is terrible wrong. We've been putting real country names in the table, and in English, but now we want to internationalize our site. That means we should avoid English words in our seed data, and instead use normalized codes that our UI layer can use and leverage existing internationalization and localization tools against. So we need to change that country table, and do so in a way to make sure we keep our existing country information correct. Lastly, we want to add a few new countries to the list as well as one more artist to the system. Thats a bunch of changes, so lets get to it!

Changes to be made:

1 - Change Country table to use codes not English names
2 - Update the table data to match above
3 - Add new countries to list
4 - Add one additional artist

So the change here is we drop the 'name' column, and replace it with a 'code' column, that is fixed as a 3 character datatype. We decided that we'd use the standard international 3 digit codes for countries, that's something we can use to wrap internationalization UI around. We also change the unique constraint that used to be on 'name' to 'code'. Everything else stays the same.

There's no other database changes, the rest will be handled in the migration so let's set our database back to version 2 (so that we have something to diff against and also migrate data from).

When the tables are small like this, you can get away with just dropping them and rebuilding everything from the ground up. Once the tables are large you will probably need to actually write correct downgrades, since that would be much more efficient.

Now we are ready to prepare the version (again):

dbic-migration -Ilib prepare --force_overwrite

We need to use the force_overwrite flag to tell DBIx::Class::DeploymentHandler that it is ok to overwrite the generated files. Since you've been making all your customizations in new files we don't have to worry about accidentally blowing away anything important. You'd expect some output like so:

Output above has been abbreviated a bit to highlight the important information. Don't worry about that "Copying Fixture Confs from ..." overwriting any of your custom changes, if there is a file in the target directory matching we just skip the copy (we always assume if the file is there that you may have made some changes you'd rather not lose).

Now, let's look at the new share/migrations/SQLite/upgrade/2-3/001-auto.sql

Again, to be brief I've only included above the new statements related to our changes to the country table. This actually seems pretty good. Lets break that out into a separate file, and add some statements to move data from the old name to the new code columns:

with the three separate INSERTs so I could properly map the English country names to the new 3 character country codes. Otherwise I've kept the rest. This again is a good example of how your diff should both change the database and alter your data in a consistent manner.

Don't forget to delete the 001-auto.sql file:

rm share/migrations/SQLite/upgrade/2-3/001-auto.sql

In order to complete our new requirements, lets create some Perl run files to add some new country codes, and one new Artist:

You might notice that the relationship names in 006-new_artist.pl don't exactly match those in our schema. As I mentioned before, this is because the $schema that is passed as the first (and only) argument to your anonymous subroutinues is NOT the schema that comes from MusicBase::Schema but instead it is generated directly from the database using DBIx::Class::Schema::Loader This is because your schema is going to change a lot, we can't rely on it always being backwardly compatible with every version of the database.

If you every get confused about what the auto generated schema looks like, you can always use the make_schema command:

## example command, don't need to run as part of the tutorial
dbic-migration -Ilib make_schema

And that will dump the current database version generated schema to share/dumped_db. You can also set a debugging %ENV variable which will dump to STDOUT the generated classes:

## example command, don't need to run as part of the tutorial
export DBIC_MIGRATION_DEBUG=1

Ok, that was a big section to get through, but we covered a lot of ground. You have seen how to handle a more complicated change set, and we added some tests and dealt with changing requirements mid task.