Custom and Private Data in Structured Logs

In Liquibase 4.23.2+, you can access custom (user-defined) data from Liquibase operations using structured logs. This allows you to collect, inspect, and analyze the information to better understand and improve your software development life cycle. Learn how to include or exclude your custom data in the default custom-log-data-file, which ships with the Liquibase installer and the artifacts.

In Liquibase 4.23.2, custom values are only added to the log if the file is explicitly defined via the CLI, defaults file, or environment variable.

  • CLI: --custom-log-data-file=liquibase.customlogdata.yaml
  • Defaults file: liquibase.customLogDataFile=my-data.yaml
  • Environment variable: LIQUIBASE_CUSTOM_LOG_DATA_FILE=liquibase.UK-data-props.yaml

In Liquibase 4.24.0+, any custom data is included in every log message. This enables you to use dashboard queries and reports that depend on these custom key:values pairs. You can set the --custom-log-data-frequency argument to ONCE or REPEATED (default) to control how much custom data is displayed in your logs. Additionally, in 4.24.0+ you can send custom data to Liquibase via environment variables on your machine (VAR_NAME=value) and via property substitution in your userMetadata file (${PROP_NAME}).

You can also use a custom metadata file to exclude the Structured Logging Keys that normally appear in every log message, such as liquibaseSchemaName and operationStart.

Create a metadata file

  1. Copy the liquibase.customlogdata.yaml file from the examples directory and save it in a secure location.
  2. Add the custom data to the relocated file as needed.
  3. If the new location is in the current working directory, no further action is needed. If not, use the --custom-log-data-file argument to specify its path.

Introduce customized data methods

Global metadata

Global metadata is general-purpose userMetadata that needs tracking every time Liquibase runs.

Global metadata rules:

  • Appears ONCE in the log (per Liquibase operation).
  • Is the output for any command execution that generates a log.
  • Is conceptually similar to the SYSTEM Structured Log Output.

Command metadata

Command metadata is userMetadata that is specific to a command.

Command MetaData rules:

  • Appears ONCE in the log per Liquibase operation of that command-type.
  • It is associated with a specific Liquibase command defined in the YAML config.
  • It is output when the associated command executes.
  • It is NOT embedded in each structured log output statement associated with the command.
    • For example, on an update, the command metadata will NOT be in the structured log output from liquibase.changelog, liquibase.executor, liquibase.* classes.

Note: You cannot exclude properties that are inside of an array.

#### USE THIS FILE TO INJECT OR EXCLUDE METADATA FROM STRUCTURED LOGS
#### on the CLI: --custom-log-data-file=liquibase.customlogdata.yaml
#### via Env var: LIQUIBASE_CUSTOM_LOG_DATA_FILE=liquibase.customlogdata.yaml
#### or other standard property setting method. For more info, see
#### https://docs.liquibase.com/structured-logging
####
## Using liquibase.userMetadata, the subset key-values
## are added to that command's MDC structured log output
## These are just examples! Please adjust before using.


## template of standalone object for all commands
liquibase.userMetadata:
  - data:
      FINDME: "example custom log data"
      teamsize: 3
      members:
        - name: danthedev
        - name: debthedba
        - name: annthearchitect


## By specifying the command, these objects will be added only
## to the logs of the update command, in this example.
liquibase.command.update:
  - data:
      awsRegion: "us-example"
      deploymentlead: "name@example.com"
      partridges:
        count: 1
        location: "a pear tree"
      turtledoves: two
      frenchhens: 3
      callingbirds: four
      goldenrings: "five"
    excludeData:
      - liquibaseTargetUrl
      - commandLabelFilter
      - commandContextFilter


## EXCLUDE data from structured logs.
## In this example, these keys will be excluded from all commands
liquibase.excludeData:
## These are the default minimum keys. Uncomment (remove the ##) to exclude these keys
## or add other keys which are not useful for your logs

## - deploymentId
## - deploymentOutcome
## - deploymentOutcomeCount
## - rowsAffected
## - changelog
## - changesetId
## - changesetAuthor
## - changesetOutcome
## - changesetCount
## - operationStart
## - operationStop
## - liquibaseCommandName
## - liquibaseSystemName
## - liquibaseSystemUser
## - liquibaseTargetUrl-
## - liquibaseVersion
## - liquibaseSchemaName
## - liquibaseCatalogName