Use --search-path
with S3
The search path is used to store the base elements of the path to Liquibase changelog files or snapshot files. changelog files (and a few others) cannot be specified by an absolute path on the file property itself, so the search path is combined with the relative path of the file's property to find these files stored remotely on S3.
For more information, see How Liquibase Finds Files: Liquibase Search Path. You can set the --search-path
property in the CLI, within the liquibase.properties
file, as an environment variable, or any other standard method for setting a property.
Use --search-path
to locate changelogs on S3
Both an S3 directory in the search path and a relative path are required to find changelog files or snapshot files. Follow the steps below to learn how to set both of these and configure them.
- Determine the S3 paths necessary to locate the changelog to use. If changelog resources, such as additional included changelogs or SQL files are in a different S3 path, you will have multiple items in the searchpath.
For example, if a changelog file is in S3 ats3://myS3Bucket/demo
and the SQL files referenced by the changelog are ins3://myS3Bucket/demo/sqlFiles
, your search path iss3://myS3Bucket/demo,s3://myS3Bucket/demo/sqlFiles
. -
To execute Liquibase commands that read changelogs located in S3, provide the searchpath and a filename or relative path to the file.
Example 1:
update-sql
command using a changelog file in S3:liquibase --search-path=s3://myS3Bucket/demo,s3://myS3Bucket/demo/sqlFiles update-sql --changelog-file=example-changelog.sql liquibase --search-path=s3://myS3bucket/demo update-sql --changelog-file=example-changelog.sql
Example 2:
update-sql
command using a relative path to the changelog file in S3:If the changelog is located in subdirectory like
s3://myS3Bucket/demo/changelogs
, when you use the searchpath defined in the previous example, then you must specify the relative path to the changelog:liquibase --search-path=s3://myS3Bucket/demo,s3://myS3Bucket/demo/sqlFiles update-sql --changelog-file=changelogs/example-changelog.sql
Note: Paths to changelogs must always be relative to preserve integrity of the Liquibase DATABASECHANGELOG tracking table.
Rules
Below are all of the rules that apply to --search-path
. Review the example use cases and instructions to ensure it is the right avenue for your project.
Multiple searchpaths
If search-path
is set, it is used and Liquibase will not look in other locations for changelog or snapshot files. Therefore, if some necessary files are located on S3 and some are local, both locations are required in the search path.
- When not specifically configured, by default
--search-path
looks for changelogs in the current working directory. - When
--search-path
is configured, only the locations explicitly defined in--search-path
are used to locate changelogs. - Multiple
--search-path
locations are comma-separated and processed in order. For example:--search-path=s3://myS3Bucket/demo,C:\Users\Me\demo
-
If multiple
--search-path
locations are specified, Liquibase may find two or more changelog files with the same name. To determine how Liquibase behaves in this case, see the "Duplicate File Mode" section of this document.Note: If the changelog references resource files in multiple directories, it may be easier to define the
--search-path
as an environment variable. This prevents the--search-path
property from making the command line unreadable. The--search-path
environment variable isLIQUIBASE_SEARCH_PATH
.
Multiple --search-path
locations example:
--search-path=s3://mybucket/path,.localfiles
Note: s3://mybucket/path
is the first location and .localfiles
is the second location that search-path
will find the files.
Duplicate File Mode
The --duplicate-file-mode
property determines how Liquibase behaves when multiple changelogs are found in the multiple locations specified in the --search-path
property. By default, an ERROR
will be thrown to halt execution, but --duplicate-file-mode
can be set to SILENT
, DEBUG
, INFO
, or WARN
instead, settings which log the issue but do not halt execution. In these cases, it uses the first changelog file found in a sequential processing of --search-path
locations.
Error Response
The ERROR
response states two files are found and Liquibase will not use either file. It will then suggest you rename the file or correct the value you specify for --search-path
, and then re-run the command.
Example: Default behavior of --duplicate-file-mode
when changelog.xml
is located in two --search-path
directories.
In this example, a changelog named changelog.xml
is located in both s3://myS3bucket/demo
and in s3://myS3Bucket/demo2
.
liquibase --search-path=s3://myS3Bucket/demo,s3://myS3Bucket/demo2 update-sql --changelog-file=changelog.xml
Liquibase console output shows an error and does not execute the update-sql
command.
Unexpected error running Liquibase: Error changelog.xml
- Caused by: Found 2 files with the path 'changelog.xml':
- s3://myS3Bucket/demo/changelog.xml
- s3://myS3Bucket/demo2/changelog.xml
Search Path:
- s3://myS3Bucket/demo/changelog.xml
- s3://myS3Bucket/demo2/changelog.xml
You can limit the search path to remove duplicates with the liquibase.searchPath setting. Or, if you KNOW these are the exact same file you can set liquibase.duplicateFileMode=WARN.
liquibase --duplicate-file-mode=ERROR --search-path=s3://myS3Bucket/demo,s3://myS3Bucket/demo2 update-sql --changelog-file=changelog.xml
results in update-sql
exiting with an ERROR
that duplicate changelog.xml
file was located.
Example: Behavior when --duplicate-file-mode
set to ERROR
and changelog.xml
is located in two search-path
directories
Although the default for --duplicate-file-mode
is ERROR
, you can explicitly configure the setting to throw an error. In this example, a changelog named changelog.xml
is located in both s3://myS3bucket/demo
and in s3://myS3Bucket/demo2
.
liquibase --duplicate-file-mode=ERROR --search-path=s3://myS3Bucket/demo,s3://myS3Bucket/demo2 update-sql --changelog-file=changelog.xml
Liquibase console output shows an error and does not execute the update-sql
command.
Unexpected error running Liquibase: Error changelog.xml
- Caused by: Found 2 files with the path 'changelog.xml':
- s3://myS3Bucket/demo/changelog.xml
- s3://myS3Bucket/demo2/changelog.xml
Search Path:
- s3://myS3Bucket/demo/changelog.xml
- s3://myS3Bucket/demo2/changelog.xml
You can limit the search path to remove duplicates with the liquibase.searchPath setting. Or, if you KNOW these are the exact same file you can set liquibase.duplicateFileMode=WARN.
Warn response
The WARN
response states two files are found and it will automatically use the first file found.
Example: Behavior when --duplicate-file-mode
set to WARN
and changelog.xml
is located in two --search-path
directories.
To explicitly ignore duplicate changelogs in the searchpath, configure --duplicate-file-mode
to WARN
. In this example, a changelog named changelog.xml
is located in both s3://myS3bucket/demo
and in s3://myS3Bucket/demo2
.
liquibase --duplicate-file-mode=WARN --search-path=s3://myS3Bucket/demo,s3://myS3Bucket/demo2 update-sql --changelog-file=changelog.xml
Liquibase console output shows a warning. The update-sql
command is executed using the changelog in s3://myS3Bucket/demo
.
Found 2 files with the path 'changelog.xml':
- s3://myS3Bucket/demo/changelog.xml
- s3://myS3Bucket/demo2/changelog.xml
Search Path:
- s3://myS3Bucket/demo/changelog.xml
- s3://myS3Bucket/demo2/changelog.xml
You can limit the search path to remove duplicates with the liquibase.searchPath setting.
To fail when duplicates are found, set liquibase.duplicateFileMode=ERROR
Choosing: s3://myS3Bucket/demo/changelog.xml