Custom and Private Data in Structured Logs

In Liquibase 4.23.2+, you can access custom (user-defined) data from Liquibase operations using structured logs. This allows you to collect, inspect, and analyze the information to better understand and improve your software development life cycle. Learn how to include or exclude your custom data in the default custom-log-data-file, which ships with the Liquibase installer and the artifacts.

In Liquibase 4.23.2, custom values are only added to the log if the file is explicitly defined via the CLI, defaults file, or environment variable.

  • CLI: --custom-log-data-file=liquibase.customlogdata.yaml
  • Defaults file: liquibase.customLogDataFile=my-data.yaml
  • Environment variable: LIQUIBASE_CUSTOM_LOG_DATA_FILE=liquibase.UK-data-props.yaml

In Liquibase 4.24.0+, any custom data is included in every log message. This enables you to use dashboard queries and reports that depend on these custom key:values pairs. You can set the --custom-log-data-frequency argument to ONCE or REPEATED (default) to control how much custom data is displayed in your logs. Additionally, in 4.24.0+ you can send custom data to Liquibase via environment variables on your machine (VAR_NAME=value) and via property substitution in your userMetadata file (${PROP_NAME}).

You can also use a custom metadata file to exclude the Structured Logging Keys that normally appear in every log message, such as liquibaseSchemaName and operationStart.

Create a metadata file

  1. Copy the liquibase.customlogdata.yaml file from the examples directory and save it in a secure location.
  2. Add the custom data to the relocated file as needed.
  3. If the new location is in the current working directory, no further action is needed. If not, use the --custom-log-data-file argument to specify its path.

Introduce customized data methods

Global metadata

Global metadata is general-purpose userMetadata that needs tracking every time Liquibase runs.

Global metadata rules:

  • Appears ONCE in the log (per Liquibase operation).
  • Is the output for any command execution that generates a log.
  • Is conceptually similar to the SYSTEM Structured Log Output.

Command metadata

Command metadata is userMetadata that is specific to a command.

Command MetaData rules:

  • Appears ONCE in the log per Liquibase operation of that command-type.
  • It is associated with a specific Liquibase command defined in the YAML config.
  • It is output when the associated command executes.
  • It is NOT embedded in each structured log output statement associated with the command.
    • For example, on an update, the command metadata will NOT be in the structured log output from liquibase.changelog, liquibase.executor, liquibase.* classes.

Note: You cannot exclude properties that are inside of an array.

Copy
liquibase.usermetadata.yaml example file
#### liquibase.usermetadata.yaml
## v1.0
## This file is used to inject or exclude data from structured logs, a Pro capability enabled by LIQUIBASE_LOG_FORMAT=JSON (or JSON_PRETTY).
## This file can be re-named and stored not in the CWD, if you specify it with --custom-log-data-file=path/to/my/renamed/customdatafile.yaml
## Learn more about Liquibase's Observability options, such as stuctured logging at https://docs.liquibase.com/structured-logging


#### GLOBAL DATA
## This is general purpose data that needs tracking every time Liquibase runs.
## ADDING: To inject custom data, create a liquibase.userMetadata object
## and the subset key-value pairs are added once to structured log output
## EXCLUDING: To exclude specific data, create a liquibase.excludeData object
## to list keys to exclude from output.


#### COMMAND-SPECIFIC DATA
## This is user-supplied data specific to a command.
## ADDING: To inject your custom data, create a liquibase.command.<commandname>.data object
## and the subset key-value pairs are added once to structured log output
## EXCLUDING: To exclude specific data, create a liquibase.command.<commandname>.excludeData object
## to list keys to exclude from output.



## These are just examples! Please adjust before using.
liquibase.userMetadata:
  - outputToLoglevel: INFO
    data:
      deploymentlead: "name@example.com"
  - outputToLogLevel: fine
    excludeData:
      - liquibaseTargetUrl
      - commandLabelFilter
      - commandContextFilter
  - outputToLogLevel: severe
    data:
      calling-birds: four
      french-hens: 3
      golden-rings: 2
      partridges:
        count: 1
        location: "a pear tree"
      turtle-doves: two


# Globally excluding these keys at all loglevels
liquibase.excludeData:
  - liquibaseCommandName
  - thread
  - changesetsUpdated.changeset


# Include data just for update command at the specified outputToLogLevel
liquibase.command.update:
  - outputToLogLevel: info
    data:
      team: "overnight-updaters"
      CiCdVersion: "6.32.1"
    excludeData:
      - thread

# Include data just for update command at the specified outputToLogLevel
liquibase.command.update-one-changeset:
  - outputToLogLevel: FINE
    data:
      reasons:
        goal: "update zipcodes"
        author: "deb the dba"
        approval: "self"
        team: "data integrity"