What are the memory limits on inspecting large schema?

Causes

You may experience Liquibase execution failures if you run resource-intensive commands on a database containing several thousand objects. This is because Liquibase commands like diff store a model of database objects in system memory before writing it to disk. If your database is very large, the memory heap reaches capacity before Liquibase can finish running the command. As a result, Java throws the OutOfMemoryError exception.

This may be more likely to occur in databases containing tens of thousands, hundreds of thousands, or millions of objects. However, the exact limit depends on the quantity, type, and complexity of objects in your database. It also depends on your database’s engine technology, custom schemas, machine processing power, and RAM (random access memory).

You may never experience these performance limitations. To estimate how they could impact your workflow, create a test database with a large, complex schema. Then, run Liquibase commands and observe your database’s behavior.

Solutions

--snapshot-filters parameter

In Liquibase Secure 4.26.0 and later, you can use the --snapshot-filters parameter to limit the types of database objects you include in your snapshot. If your database is very large, including only the objects relevant to you in the snapshot can improve the command’s performance.

For example, when you run the snapshot command in your command line, you can include only a few important objects:

liquibase snapshot --snapshot-filters=table,view

This includes only tables and views in the snapshot. For more information, see snapshot.

Java Virtual Machine settings

The Java Virtual Machine (JVM) enables your computer to run Java programs like Liquibase. You can use the JAVA_OPTS Environment Variable to modify JVM settings that affect Liquibase commands.

To increase the maximum memory heap size of the JVM, specify a new value for the JVM option -Xmx. This controls memory managed by the JVM garbage collector. Use the format -Xmx<value><unit>. For example, the following command sets maximum heap memory to 1 GB:

• Mac/Linux syntax:

• $ JAVA_OPTS="-Xmx1g" liquibase generate-changelog --changelog-file=newChangeLog.xml

• Windows syntax:

• $ set JAVA_OPTS="-Xmx1g" $ liquibase generate-changelog --changelog-file=newChangeLog.xml

Note: In addition to -Xmx (maximum heap size), you may also want to try using -Xmn (initial and maximum heap size) and -Xmn (minimum and initial heap size). See Java: Non-Standard Options and the Java Standard Edition HotSpot Virtual Machine Garbage Collection Tuning Guide.

Warning: If you set the JVM’s maximum heap memory usage too high, Liquibase may compete for resources with other applications on your system.

Hardware upgrades

If necessary, you can also upgrade your system RAM. If the JVM can access more physical memory, Liquibase can successfully run more resource-intensive commands.

For measured JVM heap and RAM sizing guidance by schema size, including a sizing matrix for snapshot, generate-changelog, diff, and diff-changelog, see Size JVM heap and RAM for diff-class commands.

Memory and heap sizing for large changeset bodies

As of version 5.2.1, Liquibase uses less memory when parsing formatted-SQL changeset bodies that contain no ${...} variable substitutions. This allows large placeholder-free changeset bodies to run successfully at modest heap sizes. For example, a 128 MB body runs without an out-of-memory error at -Xmx512m.

Keep the following in mind when working with large changeset bodies:

• Changeset bodies containing ${...} variable substitutions are unaffected by this optimization and still require heap proportional to body size. At minimum, set -Xmx to 2x the body size. A 128 MB body with variable substitutions requires -Xmx of approximately 1 GB or more.

• This optimization applies only to formatted-SQL changelogs (.sql files with the --liquibase formatted sql header). XML, YAML, and JSON changelogs use different parsers and are unaffected.

• When a changeset body exceeds 16 MB, Liquibase emits a FINE-level advisory log reporting the body size, the configured -Xmx, and whether variable substitution was applied or short-circuited. For example: Large changeset body (127 MB, -Xmx=512 MB) contains no ${...} parameter references; short-circuiting expandExpressions to avoid the OSS doubling-grow allocation. If the log reports that variable substitution was applied, review your -Xmx setting before hitting an out-of-memory error.