Troubleshoot matching specified properties in Databricks

As a Liquibase user who also uses the Databricks Delta Lake tool, you may encounter the error code [DELTA_CREATE_TABLE_WITH_DIFFERENT_PROPERTY] The specified properties do not match the existing properties when you are using the allowColumnDefaults Databricks parameter. This parameter is specifically used with the Delta Lake ALTER TABLE command when managing tables. You can set default values for columns in a Delta table schema and ensures that new rows inserted into the table will have default values for columns that aren't explicitly given a value. This is helpful when you want certain fields in your table to automatically receive a specific value when not provided.

Cause

When Liquibase Secure creates an external table from a modeled changeset, it triggers the creation of these three table properties:

delta.feature.allowColumnDefaults=supported delta.columnMapping.mode=name delta.enableDeletionVectors=true Databricks creates metadata for two of these properties in an external location, but not for allowColumnDefaults. If you delete the external table, you are not deleting the files for this table in external locations, so the metadata continues to exist. When you create this table after deleting it, the error occurs and properties will not match because Databricks will continue to look for the old, persisting metadata and not for the newly created table.

Error Message

Caused by: com.databricks.sql.transaction.tahoe.DeltaAnalysisException: [DELTA_CREATE_TABLE_WITH_DIFFERENT_PROPERTY] The specified properties do not match the existing properties at s3://databricks-test/test_table_properties.

== Specified == delta.columnMapping.mode=name delta.enableDeletionVectors=true delta.feature.allowcolumndefaults=supported

== Existing == delta.columnMapping.mode=name delta.enableDeletionVectors=true

This becomes an issue when you run liquibase update, which creates an external table in the CI/CD pipeline. If a failure occurs at any point in the pipeline, and you delete the table with the reference rather than the source containing the metadata, the file in the external location remains. In this situation, Liquibase Secure can't create a new table because Databricks is still referencing the old metadata property.

Important! You should only use this process if you can't delete the table files in the external location. Deleting the files is the easiest way to resolve this issue.