Memory Limits of Inspecting Large Schemas

When your database contains a very large number of complex objects, Liquibase may not be able to successfully execute database inspection use-cases, such as drift detection. This includes commands like diff, diff-changelog, generate-changelog, snapshot, and snapshot-reference.

Causes

You may experience Liquibase execution failures if you run resource-intensive commands on a database containing several thousand objects. This is because Liquibase commands like diff store a model of database objects in system memory before writing it to disk. If your database is very large, the memory heap reaches capacity before Liquibase can finish running the command. As a result, Java throws the OutOfMemoryError exception.

This may be more likely to occur in databases containing tens of thousands, hundreds of thousands, or millions of objects. However, the exact limit depends on the quantity, type, and complexity of objects in your database. It also depends on your database's engine technology, custom schemas, and your machine's processing power and RAM (random access memory).

Tip: You may never experience these performance limitations. To estimate the point they could impact your workflow, create a test database with a large, complex schema. Then run Liquibase commands and observe your database's behavior.

Solutions

--snapshot-filters parameter

In Liquibase Pro 4.26.0 and later, you can use the --snapshot-filters parameter to limit the types of database objects you include in your snapshot. If your database is very large, including only the objects relevant to you in the snapshot can improve the command's performance.

For example, when you run the snapshot command in your command line, you can include only a few important objects:

liquibase snapshot --snapshot-filters=table,view

This includes only tables and views in the snapshot. For more information, see snapshot.

Java Virtual Machine settings

The Java Virtual Machine (JVM) enables your computer to run Java programs like Liquibase. You can use the JAVA_OPTS Environment Variable to modify JVM settings that affect Liquibase commands.

To increase the maximum memory heap size of the JVM, specify a new value for the JVM option -Xmx. This controls memory managed by the JVM garbage collector. Use the format -Xmx<value><unit>. For example, the following command sets maximum heap memory to 1 GB:

  • Mac/Linux syntax:
    • $ JAVA_OPTS="-Xmx1g"
    $ liquibase generate-changelog --changelog-file=newChangeLog.xml
  • Windows syntax:
    • $ set JAVA_OPTS="-Xmx1g"
    $ set liquibase generate-changelog --changelog-file=newChangeLog.xml

Note: In addition to -Xmx (maximum heap size), you may also want to try using -Xmn (initial and maximum heap size) and -Xmn (minimum and initial heap size). See Java: Non-Standard Options and the Java Standard Edition HotSpot Virtual Machine Garbage Collection Tuning Guide.

Warning: If you set the JVM's maximum heap memory usage too high, Liquibase may compete for resources with other applications on your system.

Hardware upgrades

If necessary, you can also upgrade your system RAM. If the JVM can access more physical memory, Liquibase can successfully run more resource-intensive commands.

To determine approximately how much more RAM you need to run a particular command, estimate your current memory limit:

  1. Compare your current schema to smaller schemas with similarly complex objects.
  2. Run a Liquibase database inspection command like snapshot on the smaller schemas.
  3. If it succeeds on a small schema but fails on a slightly larger schema, the memory limit must be somewhere in between.
  4. Use a hardware monitoring tool to see how much RAM Liquibase uses when running the command on each schema.

For example, assume big-schema has 1 million objects of a given amount of complexity, and snapshot always returns an OutOfMemoryError. If you can successfully run snapshot on small-schema (containing 700,000 similar objects) but not medium-schema (containing 800,000 similar objects), your memory limit for snapshot is approximately 750,000 objects. In this case, you would need ~33% more RAM to run snapshot on big-schema.

Note: This method may be inaccurate if your test schemas do not have similar objects and architectures. It may also be inaccurate if you compare test results from different databases or machines.

Related links