What Databricks features does Liquibase support?

Databricks SQL versions verified

• Cloud

• AWS

• Azure

• Google Cloud Platforms

Getting started tutorial

To learn how to install, configure, and use the Liquibase Databricks extension with your instance of Databricks, see Connect Liquibase with Databricks SQL. This page contains driver download links, guidance on permissions, and a sample changelog for a test deployment.

Features

Supported Liquibase Secure features

• Policy Checks: automatically analyze your changelogs for desired format and behavior to increase deployment success rates and uphold security best practices.

• Secrets Management: keep your authentication data secure by integrating it with third-party secrets vaults.

• Structured Logging: improve your database observability by easily reading Liquibase data in your favorite analytics tool.

• Operation Reports: generate reports of operations you perform on your database.

• Flow Files: create repeatable, portable, and platform-independent Liquibase workflows to run in any of your CI/CD tools.

• DATABASECHANGELOGHISTORY table (DBCLH): record a history of all changes you make to the database, including changes that are not tracked by the DATABASECHANGELOG table (DBCL).

• Remote file access: centralize file management with AWS S3 to build a reusable repository of Liquibase files you can update and retrieve.

• Targeted rollback: avoid collateral damage by specifying which changesets in your changelog to undo.

• Stored Logic: capture stored logic objects like functions

Supported Change Types

Databricks-specific Change Types available with the Liquibase Open Source Databricks extension and Liquibase Secure Databricks extension:

• alterCluster: alter a clustered table

• alterTableProperties: alter existing properties on a table

• alterViewProperties: alter existing properties on a view

• analyzeTable: analyze a table to improve query performance

• optimizeTable: optimize table layout to improve performance

• vacuumTable: remove unused files in a table directory

Databricks-specific Change Types that require the Liquibase Secure Databricks extension:

• alterVolume: alter a volume

• cloneTable: creates a table that clones the metadata and/or row data an existing table

• createVolume: create a volume

• dropVolume: drop a volume

• restoreTable: restore a table to an earlier state

• syncIdentityColumn: synchronize identity column metadata with actual data

Liquibase Open Source Change Types that accept Databricks-specific attributes or sub-tags:

createTable: create a table. Databricks sub-tag: extendedTableProperties: specify additional properties on a table, including clusters and partitions

createView: create a view on a Databricks table. Databricks attribute: tblProperties: similar to extendedTableProperties, but for a view

Liquibase Change Types:

• addColumn

• addForeignKeyConstraint

• addLookupTable

• addNotNullConstraint

• addPrimaryKey

• createFunction (requires Liquibase Secure extension). Both SQL and Python functions are supported.

• createTable

• delete

• dropAllForeignKeyConstraints

• dropColumn

• dropForeignKeyConstraint

• dropFunction (requires Liquibase Secure extension).

• dropNotNullConstraint

• dropPrimaryKey

• dropTable

• executeCommand

• insert

• loadData

• loadUpdateData

• mergeColumns

• modifyDataType

• modifySql

• renameColumn

• renameTable

• setColumnRemarks

• setTableRemarks

• sql

• sqlFile

Supported parameters

You can use all normal Liquibase parameters with Databricks. The Liquibase Secure extension for Databricks offers additional parameters for more advanced functionality.

Note: The following parameters, including databricks-catalog and databricks-schema, only work with OIDC authentication. They will not work for token authentication.

To configure Databricks-specific behavior, set the following Liquibase parameters in the CLI, in flow files, in your liquibase.properties file, or as environment variables:

• --databricks-auth-mechanism

• --databricks-catalog

• --databricks-http-path

• --databricks-oauth-auth-flow

• --databricks-oauth-client-id

• --databricks-oauth-client-secret

• --databricks-schema

For more information, see Liquibase parameters for Databricks.

Supported commands

You can use the Liquibase CLI commands or Liquibase Maven goals with Databricks:

• Update Commands

• Rollback Commands

• Init Commands

• Change Tracking Commands

• Database Inspection Commands

• Utility Commands

• Flow Commands

• Policy Checks Commands

Row data capture

Use row data capture when you baseline an existing Databricks database, so the reference and seed data your tables already hold becomes part of the changelog alongside the schema.

When you run generate-changelog or diff-changelog with --should-snapshot-data=true, Liquibase captures the existing rows in your Databricks tables as insert changesets. If you also set --object-changelogs=all or --object-changelogs=dml, Liquibase writes those changesets to objects/inserts/ sub-changelogs, one file per table, and includes them from the master changelog. With --object-changelogs=table, the insert changesets stay inline in the generated changelog.

Data type handling

For general information about how Liquibase handles data types in Databricks, see Liquibase Data Type Handling. However, be aware that Liquibase does not support the change types listed in the Limitations section.

Limitations

Liquibase does not support the following Change Types:

• addAutoIncrement

• addUniqueConstraint

• createIndex

• createPackage

• createPackageBody

• createProcedure

• createSequence

• createTrigger

• disableCheckConstraint

• dropPackage

• dropPackageBody

• dropUniqueConstraint

• enableCheckConstraint

• modifyDataType

• renameTable

• You can deploy changes to Databricks using formatted SQL, YAML, JSON, and XML changelogs, but not unformatted SQL changelogs. To write SQL directly, you can use the sql and sqlFile Change Types.

• Liquibase does not include the values of tableFormat and partitionColumns in files generated by the database inspection commands diff-changelog and generate-changelog. Databricks cannot add new partition columns to an existing table, nor can it change the table's format or location. Running these attributes on existing tables would cause failures, so Liquibase omits them from changesets it generates. However, the outputs of diff and snapshot do include tableFormat and partitionColumns.

• Liquibase does not include all auto-increment (identity) information in files generated by diff-changelog. This information is omitted when a table has the same name in both the source and target databases, but different values for autoIncrement. Also, the addAutoIncrement Change Type is not supported for Databrick. However, files generated by the diff command do include auto-increment information even when the tables have different values for autoIncrement.

• Databricks does not support directly adding a new column with a default value to a table that already contains columns with default values. In Liquibase Databricks 1.4.0, you cannot use the addDefaultValue Change Type in an YAML, JSON, or XML changelog to add a default value to an existing Databricks table column. Instead, you must specify the SQL queries for addDefaultValue in a Formatted SQL changelog or use the sql or sqlFile Change Types in a YAML, JSON, or XML changelog.