Use search-path with S3

The search path is used to store the base elements of the path to Liquibase changelog files or snapshot files. changelog files (and a few others) cannot be specified by an absolute path on the file property itself, so the search path is combined with the relative path of the file's property to find these files stored remotely on S3.

For more information, see How Liquibase Finds Files: Liquibase Search Path. You can set the search-path property in the CLI, within the liquibase.properties file, as an environment variable, or any other standard method for setting a property.

Use search-path to locate changelogs on S3

Both an S3 directory in the search path and a relative path are required to find changelog files or snapshot files. Follow the steps below to learn how to set both of these and configure them.

  1. Determine the S3 paths necessary to locate the changelog to use. If changelog resources, such as additional included changelogs or SQL files are in a different S3 path, you will have multiple items in the searchpath.

    For example, if a changelog file is in S3 at s3://myS3Bucket/demo and the SQL files referenced by the changelog are in s3://myS3Bucket/demo/sqlFiles, your search path is s3://myS3Bucket/demo,s3://myS3Bucket/demo/sqlFiles.

  2. To execute Liquibase commands that read changelogs located in S3, provide the searchpath and a filename or relative path to the file.

    Example 1: update-sql command using a changelog file in S3:

    liquibase --search-path=s3://myS3Bucket/demo,s3://myS3Bucket/demo/sqlFiles update-sql --changelog-file=example-changelog.sql
liquibase --search-path=s3://myS3bucket/demo update-sql --changelog-file=example-changelog.sql

    Example 2: update-sql command using a relative path to the changelog file in S3:

    If the changelog is located in subdirectory like s3://myS3Bucket/demo/changelogs, when you use the searchpath defined in the previous example, then you must specify the relative path to the changelog:

    liquibase --search-path=s3://myS3Bucket/demo,s3://myS3Bucket/demo/sqlFiles update-sql --changelog-file=changelogs/example-changelog.sql

    3. Note: Paths to changelogs must always be relative to preserve integrity of the Liquibase DATABASECHANGELOG tracking table.

Rules

Below are all of the rules that apply to search-path. Review the example use cases and instructions to ensure it is the right avenue for your project.

Multiple searchpaths

If search-path is set, it is used and Liquibase will not look in other locations for changelog or snapshot files. Therefore, if some necessary files are located on S3 and some are local, both locations are required in the search path.

  • When not specifically configured, by default search-path looks for changelogs in the current working directory.
  • When search-path is configured, only the locations explicitly defined in search-path are used to locate changelogs.
  • Multiple search-path locations are comma-separated and processed in order. For example:
    --search-path=s3://myS3Bucket/demo,C:\Users\Me\demo

  • If multiple search-path locations are specified, Liquibase may find two or more changelog files with the same name. To determine how Liquibase behaves in this case, see the "Duplicate File Mode" section of this document.

    Note: If the changelog references resource files in multiple directories, it may be easier to define the search-path as an environment variable. This prevents the search-path property from making the command line unreadable. The search-path environment variable is LIQUIBASE_SEARCH_PATH.

Multiple search-path locations example:

--search-path=s3://mybucket/path,.localfiles

Note: s3://mybucket/path is the first location and .localfiles is the second location that search-path will find the files.

Duplicate File Mode

The duplicate-file-mode property determines how Liquibase behaves when multiple changelogs are found in the multiple locations specified in the search-path property. By default, an ERROR will be thrown, but duplicate-file-mode can be set to WARN instead. In which case, the command will continue, using the first changelog file found in a sequential processing of search-path locations.

Error Response

The ERROR response states two files are found and Liquibase will not use either file. It will then suggest you rename the file or correct the value you specify for search-path, and then re-run the command.

Example: Default behavior of duplicate-file-mode when changelog.xml is located in two search-path directories.

In this example, a changelog named changelog.xml is located in both s3://myS3bucket/demo and in s3://myS3Bucket/demo2.

liquibase --search-path=s3://myS3Bucket/demo,s3://myS3Bucket/demo2 update-sql --changelog-file=changelog.xml

Liquibase console output shows an error and does not execute the update-sql command.

Copy 
Unexpected error running Liquibase: Error changelog.xml
  - Caused by: Found 2 files with the path 'changelog.xml':
    - s3://myS3Bucket/demo/changelog.xml
    - s3://myS3Bucket/demo2/changelog.xml
  Search Path:
    - s3://myS3Bucket/demo/changelog.xml
    - s3://myS3Bucket/demo2/changelog.xml
  You can limit the search path to remove duplicates with the liquibase.searchPath setting. Or, if you KNOW these are the exact same file you can set liquibase.duplicateFileMode=WARN.

liquibase --duplicate-file-mode=ERROR --search-path=s3://myS3Bucket/demo,s3://myS3Bucket/demo2 update-sql --changelog-file=changelog.xml results in update-sql exiting with an ERROR that duplicate changelog.xml file was located.

Example: Behavior when duplicate-file-mode set to ERROR and changelog.xml is located in two search-path directories

Although the default for duplicate-file-mode is ERROR, you can explicitly configure the setting to throw an error. In this example, a changelog named changelog.xml is located in both s3://myS3bucket/demo and in s3://myS3Bucket/demo2.

liquibase --duplicate-file-mode=ERROR --search-path=s3://myS3Bucket/demo,s3://myS3Bucket/demo2 update-sql --changelog-file=changelog.xml

Liquibase console output shows an error and does not execute the update-sql command.

Copy 
Unexpected error running Liquibase: Error changelog.xml
  - Caused by: Found 2 files with the path 'changelog.xml':
    - s3://myS3Bucket/demo/changelog.xml
    - s3://myS3Bucket/demo2/changelog.xml
  Search Path:
    - s3://myS3Bucket/demo/changelog.xml
    - s3://myS3Bucket/demo2/changelog.xml
  You can limit the search path to remove duplicates with the liquibase.searchPath setting. Or, if you KNOW these are the exact same file you can set liquibase.duplicateFileMode=WARN.

Warn response

The WARN response states two files are found and it will automatically use the first file found.

Example: Behavior when duplicate-file-mode set to WARN and changelog.xml is located in two search-path directories.

To explicitly ignore duplicate changelogs in the searchpath, configure duplicate-file-mode to WARN. In this example, a changelog named changelog.xml is located in both s3://myS3bucket/demo and in s3://myS3Bucket/demo2.

liquibase --duplicate-file-mode=WARN --search-path=s3://myS3Bucket/demo,s3://myS3Bucket/demo2 update-sql --changelog-file=changelog.xml

Liquibase console output shows a warning. The update-sql command is executed using the changelog in s3://myS3Bucket/demo.

Copy 
Found 2 files with the path 'changelog.xml':
- s3://myS3Bucket/demo/changelog.xml
- s3://myS3Bucket/demo2/changelog.xml
Search Path:
- s3://myS3Bucket/demo/changelog.xml
- s3://myS3Bucket/demo2/changelog.xml
You can limit the search path to remove duplicates with the liquibase.searchPath setting.
To fail when duplicates are found, set liquibase.duplicateFileMode=ERROR
Choosing: s3://myS3Bucket/demo/changelog.xml