Smooth Database Changes in Blue-Green Deployments

Blue-green deployment is a technique of releasing changes to a project by gradually transferring users to a new version. It provides an isolation between the current “blue” environment and the new “green” environment. Blue nodes are sequentially upgraded to the new “green” version, so for the entire time of deployment we have both environments working simultaneously. It’s quite a challenge to keep both versions running smoothly and to keep downtime as short as possible, with getting close to the mythical zero-downtime.

Changes in an app logic are causing changes in Django models and in the database structure. Therefore, it’s really important to use the right strategy to make the new version backward compatible from the database perspective. Luckily for us, the migrations framework makes it possible in Django.

This article will discuss:

• How migrations work?
• How to use advanced migration tools to make database transitions in the most efficient way?
• How does migration fit into the blue-green strategy?

Let’s start with a brief introduction to the migrations framework.

Migrations framework

The migrations framework is a way of propagating changes in Django models to the database. It makes web development with Django more accessible because, as developers, we no longer need to know, write, and maintain SQL statements with data definitions (known as DDL statements - Data Definition Language). Believe me, maintaining SQL statements can be really painful. Luckily for us, we don’t have to do that anymore.

Let’s explore how it works.

Workflow

The standard workflow for changing our data definitions has three steps:

• First, make a change in a model definition. For example, we can add a new field called price to the Book model:

# bookstore/models.py

from django.db import models


class Font(models.Model):
    name = models.CharField(max_length=255)


class Author(models.Model):
    name = models.CharField(max_length=255)


class Book(models.Model):
    title = models.CharField(max_length=1023)
    description = models.TextField()
    pages = models.IntegerField()
    isbn = models.IntegerField()
    authors = models.ManyToManyField("Author")
    font = models.ForeignKey(
        "Font", on_delete=models.CASCADE, null=True
    )
    # ↓ New field ↓
    price = models.DecimalField(
        max_digits=8, decimal_places=2, null=True
    )

• Second, run the makemigrations command to generate a new migration file:

python3 manage.py makemigrations

Migrations for 'bookstore':
  bookstore/migrations/0002_book_price.py
    - Add field price to book

Migration files describe what kind of changes are needed in our database structure and what kind of changes has been made in model definitions. That’s what it looks for adding a new field:

# bookstore/migrations/0002_book_price.py

from django.db import migrations, models


class Migration(migrations.Migration):
   dependencies = [
       ("bookstore", "0001_initial"),
   ]

   operations = [
       migrations.AddField(
           model_name="book",
           name="price",
           field=models.DecimalField(
               decimal_places=2, max_digits=8, null=True
           ),
       ),
   ]

Migration files contain:
1. dependencies - list of migrations it depends on, e.g. the previous migration name.
2. operations - list of operations to perform, in our case it is AddField(). The Django documentation describes all kind of core operations.
3. initial - whether the migration is the first initial migration of an app.

• The last step is to propagate changes into our database by running the migrate command.

python3 manage.py migrate

Operations to perform:
 Apply all migrations: admin, auth, bookstore, contenttypes, sessions
Running migrations:
 Applying bookstore.0002_book_price... OK

We now have a new column in the Book table 🚀 .

Fortunately, there are many database operations that are backward compatible and don’t require any special treatment during the blue-green deployments. Django handles them really efficiently in the standard migrations flow, it includes

• adding a model (table in the database),
• adding a nullable field (column in the database),
• removing an index, or
• removing a constraint.

Unfortunately, some database transformations are backward incompatible and we need to choose the right strategy and tools to keep them smooth from deployment perspective, it includes

• removing a field,
• removing a model, or
• adding a non-nullable field.

Let’s check what advanced tools are built into Django and how we can use them.

Separate database and project state

What we want to do is to separate model definitions from the corresponding database structure during the blue-green deployments. We want to ensure that removed fields (or models) are no longer used on new “green” nodes, while at the same time the required database columns (or tables) are available for old “blue” nodes.

Django’s answer to this problem is a special and extremely powerful migration operation called SeparateDatabaseAndState(). It allows to separate database and project state, so to separate changes that should be made in the database structure from the changes that are recognized by Django as being made in the state of models. It allows us to persuade Django that project operations were made as it expects and in the same time no SQL statements are issued on the database.

SeparateDatabaseAndState(
    database_operations=[...],
    state_operations=[...],
)

It accepts two lists of operations:

• database_operations - list of operations to apply to the database,
• state_operations - list of operations to apply to the project state.

Let’s find out how we can use it in practice for removing a field and a model.

Full example

We need an existing or new Django project. Here are some great resources for getting started with Django or deploying your Django app to Fly.io.

With a project ready, let’s get started!

Suppose that we have a bookstore site, that provides various information about available books. In our example, we’ll use the same models (Font, Author, and Book) as defined in the Workflow paragraph. What if, at some point, we realize that collecting information about fonts used in books are difficult and unnecessary. It’s not important for our clients, so we would like to simplify our models definition by removing the Font model and the Book.font field.

The following steps will show you how we can perform this potentially backward incompatible transition in a blue-green deployment by using the SeparateDatabaseAndState() operation.

• First step is to remove all logic related to the Font from our project.
• Secondly, remove the Font model and the Book.font field:

--- a/bookstore/models.py
+++ b/bookstore/models.py
@@ -1,10 +1,6 @@
 from django.db import models


-class Font(models.Model):
-    name = models.CharField(max_length=255)
-
-
 class Author(models.Model):
     name = models.CharField(max_length=255)

@@ -15,9 +11,6 @@ class Book(models.Model):
     pages = models.IntegerField()
     isbn = models.IntegerField()
     authors = models.ManyToManyField("Author")
-    font = models.ForeignKey(
-        "Font", on_delete=models.CASCADE, null=True
-    )
     price = models.DecimalField(
         max_digits=8, decimal_places=2, null=True
     )

• Next, run the makemigrations command to generate a new migration file:

python3 manage.py makemigrations

Migrations for 'bookstore':
  bookstore/migrations/0003_remove_book_font_delete_font.py
    - Remove field font from book
    - Delete model Font

makemigrations generates a new migration with two operations: RemoveField() and DeleteModel():

# bookstore/migrations/0003_remove_book_font_delete_font.py

from django.db import migrations


class Migration(migrations.Migration):
    dependencies = [
        ("bookstore", "0002_book_price"),
    ]

    operations = [
        migrations.RemoveField(
            model_name="book",
            name="font",
        ),
        migrations.DeleteModel(name="Font"),
    ]

Both operations are backward incompatible and deploying this migration as it is would break the old “blue” environments, because the underlying table and column would be deleted. Using the sqlmigrate command we can check SQL statements that will be issued (output for PostgreSQL):

python3 manage.py sqlmigrate bookstore 0003_remove_book_font_delete_font

BEGIN;
--
-- Remove field font from book
--
ALTER TABLE "bookstore_book" DROP COLUMN "font_id" CASCADE;
--
-- Delete model Font
--
DROP TABLE "bookstore_font" CASCADE;
COMMIT;

These operations describe changes that we made in models, so we can wrap them with the SeparateDatabaseAndState() operation by moving to the state_operations, at the same time leaving database_operations empty to avoid changes in the database structure, like so:

# bookstore/migrations/0003_remove_book_font_delete_font.py

from django.db import migrations


class Migration(migrations.Migration):
    dependencies = [
        ("bookstore", "0002_book_price"),
    ]

    operations = [
        migrations.SeparateDatabaseAndState(
            # Operations in the project state.
            state_operations=[
                migrations.RemoveField(
                    model_name="book",
                    name="font",
                ),
                migrations.DeleteModel(name="Font"),
            ],
            # No changes in the database.
            database_operations=[],
        ),
    ]

Using the sqlmigrate command again confirms that no SQL statements will be issued:

python3 manage.py sqlmigrate bookstore 0003_remove_book_font_delete_font

BEGIN;
--
-- Custom state/database change combination
--
-- (no-op)
COMMIT;

We can now safely deploy our new “green” version to all instances in a fully backward compatible manner from a database perspective, as nothing has changed in the database structure 🎉 .

Once the new version is deployed to all instances and when we are sure that the underlying table and column are unused, we can actually remove them from the database.

To do this, we need a new blank migration file that will be structured like other migration files. To generate such a file, use the --empty flag to the makemigrations command. In order to follow good practice, and to avoid migration names based on timestamp, I’d recommend to also pass the --name flag, e.g. "remove_book_font_delete_font_from_db" which makes it clear what this migrations will do:

python3 manage.py makemigrations bookstore \
--empty \
--name remove_book_font_delete_font_from_db

Migrations for 'bookstore':
  bookstore/migrations/0004_remove_book_font_delete_font_from_db.py

The dependencies are already filled and we have an empty list of operations ready to be use:

# bookstore/migrations/0004_remove_book_font_delete_font_from_db.py

from django.db import migrations


class Migration(migrations.Migration):
    dependencies = [
        ("bookstore", "0003_remove_book_font_delete_font"),
    ]

    operations = []

We want to perform the same operations, but this time in the database, not in the project state. Therefore, we will use SeparateDatabaseAndState() again but with SQL statements that describe RemoveField() and DeleteModel() operations in the database_operations and an empty list of state_operations. The modified migration file 0004_remove_book_font_delete_font_from_db.py should look like this:

# bookstore/migrations/0004_remove_book_font_delete_font_from_db.py

from django.db import migrations


class Migration(migrations.Migration):
    dependencies = [
        ("bookstore", "0003_remove_book_font_delete_font"),
    ]

    operations = [
        migrations.SeparateDatabaseAndState(
            # No changes in the project.
            state_operations=[],
            # Operations to be performed on the database.
            database_operations=[
                # SQL statement for RemoveField().
                migrations.RunSQL(
                    sql=(
                        'ALTER TABLE "bookstore_book" '
                        'DROP COLUMN "font_id" CASCADE;'
                    ),
                ),
                # SQL statement for DeleteModel().
                migrations.RunSQL(
                    sql='DROP TABLE "bookstore_font" CASCADE;',
                ),
            ],
        ),
    ]

We can now safely deploy our latest “green” version to all instances in a fully backward compatible manner as the deleted table and column are no longer used by any node after the previous deployment 💙 → 💚. Voilà ✨

Closing Thoughts

Separating database and project states gives us highly flexible pattern for propagating database changes. It is not only handy for blue-green deployments, but it also allows for describing potentially not feasible transitions in the most efficient way. Frequently without any data migration, without any intermediate steps, atomic, and even reversible. Some use cases are:

• changing a ManyToManyField to use a through model,
• changing a custom through model to auto-generated intermediate table,
• moving models between apps, etc.

Try it and share!

Remember to check out the new and shiny bluegreen deployment strategy on Fly.io!