At its Information + AI Summit in the present day, Databricks introduced that it’s open sourcing Unity Catalog, the metadata catalog that governs how customers and compute engines can entry information. Coming off of final week’s information round Apache Iceberg, the transfer marks an essential shift for Databricks because it seeks to keep up momentum as clients more and more demand open lakehouse platforms.
Databricks unveiled Unity Catalog again in 2021 as a strategy to govern and safe entry to information saved in Delta, the desk format that Databricks created in 2017 because the linchpin of its lakehouse technique. It has remained a proprietary product at Databricks since.
However lately, a competing desk format, Apache Iceberg, has gained momentum within the massive information ecosystem. Databricks addressed Iceberg’s rise final week with the deliberate acquisition of Tabular, the lakehouse firm based by Iceberg’s creator. Databricks’ technique is to progressively transfer the Iceberg and Delta specs nearer collectively over time, thereby eliminating the variations between them.
That left the common-or-garden metadata catalog because the final piece standing between clients and their dream of a very open information lakehouse. Databricks’ rival, Snowflake, addressed the potential lock-in of the metadata catalog final week with the launch of Polaris, which relies on Iceberg’s REST-based API. The corporate tells Datanami that it plans to donate the Polaris challenge to open supply, seemingly the Apache Software program Basis, inside 90 days.
That left the still-proprietary Unity Catalog because the odd-man out on the metadata catalog layer, simply as a brand new period of open lakehouses abruptly arrives. To handle that strategic shift available in the market, Databricks determined to open supply Unity Catalog.
The transfer creates the “USB” for information entry, Databricks CEO Ali Ghodsi mentioned throughout his keynote handle at Databricks’ Information + AI Summit in San Francisco.
“All of the silos that you simply had earlier than, they’ll simply entry one copy of the information that’s in a standardized USB format beneath your possession,” Ghodsi mentioned. “It goes by way of one governance layer that’s simply standardized–that’s Unity Catalog–for your whole information.”
Unity Catalog beforehand supported Delta and Iceberg, along with Apache Hudi, one other open desk format, through Databricks’ Delta Lake UniForm format. Actually, Unity Catalog additionally helps Iceberg’s REST-based API, Ghodsi identified.
“We mainly standardized the information layer and the safety layer so that you simply personal your information and every little thing goes by way of these open interfaces,” he mentioned. “And I feel that’s going to be superior for the group, for everyone in right here. As a result of we simply have far more use instances. We’re going to have the ability to do way more innovation, and we’ll simply broaden this marketplace for all people concerned.”
Databricks CEO Ali Ghodsi introduced the open sourcing of Unity Catalog at Information + AI Summit, June 12, 2024
Databricks clients applauded the transfer, together with AT&T and Nasdaq.
“With the announcement of Unity Catalog’s open sourcing, we’re inspired by Databricks’ step to make lakehouse governance and metadata administration attainable by way of open requirements,” mentioned Matt Dugan, AT&T’s vice chairman for information platforms. “The pliability to make the most of interoperable instruments with our information and AI property, with constant governance, is core to the AT&T information platform technique.”
“Databricks’ choice to open supply Unity Catalog offers an answer that helps get rid of information silos and we stay up for additional scaling our platform, enhancing our governance, and modernizing our information functions as we proceed to ship for our purchasers,” mentioned Lenny Rosenfeld, Nasdaq’s vice chairman of capital entry platforms.
It’s not clear what open supply basis Databricks will select for Unity Catalog OSS, nor what the timeline might be. Beforehand, Databricks has chosen The Linux Basis to open supply numerous internally developed merchandise, together with Delta and MLFlow.
Unity Catalog might be posted to Github on Thursday throughout Databricks’ CTO Matei Zaharai keynote at Information + AI Summit, the corporate mentioned.
Associated Gadgets:
All Eyes on Databricks as Information + AI Summit Kicks Off
Databricks Nabs Iceberg-Maker Tabular to Spawn Desk Uniformity
Snowflake Embraces Open Information with Polaris Catalog
