We’re excited to announce the Public Preview of LakeFlow Join for SQL Server, Salesforce, and Workday. These ingestion connectors allow easy and environment friendly ingestion from databases and enterprise apps—powered by incremental information processing and sensible optimizations beneath the hood. LakeFlow Join can be native to the Knowledge Intelligence Platform, so it gives each serverless compute and Unity Catalog governance. In the end, this implies organizations can spend much less time shifting their information and extra time getting worth from it.
Extra broadly, this can be a key step in direction of realizing the way forward for information engineering on Databricks with LakeFlow: the unified resolution for ingestion, transformation and orchestration that we introduced at Knowledge + AI Summit. LakeFlow Join will work seamlessly with LakeFlow Pipelines for transformation and LakeFlow Jobs for orchestration. Collectively, these will allow prospects to ship brisker and higher-quality information to their companies.
Challenges in information ingestion
Organizations have a variety of knowledge sources: enterprise apps, databases, message buses, cloud storage, and extra. To handle the nuances of every supply, they typically construct and preserve customized ingestion pipelines, which introduces a number of challenges.
- Advanced configuration and upkeep: It’s troublesome to connect with databases, particularly with out impacting the supply system. It’s additionally exhausting to be taught and sustain with ever-changing utility APIs. Due to this fact, customized pipelines require a number of effort to construct, optimize, and preserve—which may, in flip, restrict efficiency and enhance prices.
- Dependencies on specialised groups: Given this complexity, ingestion pipelines typically require extremely expert information engineers. Because of this information customers (e.g., HR analysts, and monetary planners) depend upon specialised engineering groups, thus limiting productiveness and innovation.
- Patchwork options with restricted governance: With a patchwork of pipelines, it’s exhausting to construct governance, entry management, observability, and lineage. This opens the door to safety dangers and compliance challenges, in addition to difficulties in troubleshooting any points.
LakeFlow Join: easy and environment friendly ingestion for each workforce
LakeFlow Join addresses these challenges in order that any practitioner can simply construct incremental information pipelines at scale.
LakeFlow Join is easy to configure and preserve
To start out, the connectors take as little as only a few steps to arrange. Furthermore, when you’ve arrange a connector, it’s totally managed by Databricks. This lowers the prices of upkeep. It additionally signifies that ingestion not requires specialised data—and that information could be democratized throughout your group.
“The Salesforce connector was easy to arrange and gives the flexibility to sync information to our information lake. This has saved quite a lot of improvement time and ongoing help time making our migration sooner”
— Martin Lee, Expertise Lead Software program Engineer, Ruffer
LakeFlow Join is environment friendly
Below the hood, LakeFlow Join pipelines are constructed on Delta Stay Tables, that are designed for environment friendly incremental processing. Furthermore, most of the connectors learn and write solely the info that’s modified within the supply system. Lastly, we leverage Arcion’s source-specific expertise to optimize every connector for efficiency and reliability whereas additionally limiting affect on the supply system.
As a result of ingestion is simply step one, we don’t cease there. You may as well assemble environment friendly materialized views that incrementally remodel your information as it really works its method by the medallion structure. Particularly, Delta Stay Tables can course of updates to your views incrementally—solely updating the rows that want to alter slightly than totally recomputing all rows. Over time, this will considerably enhance the efficiency of your transformations, which in flip makes your end-to-end ETL pipelines simply that rather more environment friendly.
“The connector enhances our skill to switch information by offering a seamless and sturdy integration between Salesforce and Databricks. […] The time required to extract and put together information has been diminished from roughly 3 hours to only half-hour”
— Amber Howdle-Fitton, Knowledge and Analytics Supervisor, Kotahi
LakeFlow Join is native to the Knowledge Intelligence Platform
LakeFlow Join is totally built-in with the remainder of your Databricks tooling. Like the remainder of your information and AI property, it is ruled by Unity Catalog, powered by Delta Stay Tables utilizing serverless compute, and orchestrated with Databricks Workflows. This allows options like unified monitoring throughout your ingestion pipelines. Furthermore, as a result of it’s all a part of the identical platform, you possibly can then use Databricks SQL, AI/BI and Mosaic AI to get essentially the most out of your information.
”With Databricks’ new LakeFlow Connector for SQL Server, we will eradicate […] middleman merchandise between our supply database and Databricks. This implies sooner information ingestion, diminished prices, and fewer effort spent configuring, sustaining, and monitoring third-party CDC options. This function will drastically profit us by streamlining our information pipeline.”
— Kun Lee, Senior Director Database Administrator, CoStar
An thrilling LakeFlow roadmap
The primary wave of connectors can create SQL Server, Salesforce, and Workday pipelines through API. However this Public Preview is just the start. Within the coming months, we plan to start Personal Previews of connectors to extra information sources, equivalent to:
- ServiceNow
- Google Analytics 4
- SharePoint
- PostgreSQL
- SQL Server on-premises
The roadmap additionally features a deeper function set for every connector. This may occasionally embody:
- UI for connector creation
- Knowledge lineage
- SCD kind 2
- Strong schema evolution
- Knowledge sampling
Extra broadly, LakeFlow Join is just the primary element of LakeFlow. Later this 12 months, we plan to preview LakeFlow Pipelines for transformation and LakeFlow Jobs for orchestration—the evolution of Delta Stay Tables and Workflows, respectively. As soon as they’re out there, they won’t require any migration. One of the best ways to organize for these new additions is to begin utilizing Delta Stay Tables and Workflows right this moment.
Getting began with LakeFlow Join
SQL Server connector: Helps ingestion from Azure SQL Database and AWS RDS for SQL Server, with incremental reads that use change information seize (CDC) and alter monitoring expertise. Study extra in regards to the SQL Server Connector.
Salesforce connector: Helps ingestion from Salesforce Gross sales Cloud, permitting you to hitch these CRM insights with information within the Knowledge Intelligence Platform to ship extra insights and extra correct predictions. Study extra in regards to the Salesforce connector.
Workday connector: Helps ingestion from Workday Experiences-as-a-Service (RaaS), permitting you to investigate and enrich your stories. Study extra in regards to the Workday connector.
“The Salesforce connector supplied in LakeFlow Join has been essential for us, enabling direct connections to our Salesforce databases and eliminating the necessity for an extra paid intermediate service.”
— Amine Hadj-Youcef, Answer Architect, Engie
To get entry to the preview, contact your Databricks account workforce.
Be aware that LakeFlow Join makes use of serverless compute for Delta Stay Tables. Due to this fact:
- Serverless compute have to be enabled in your account (see how to take action for Azure or AWS, and see a listing of serverless-enabled areas for Azure or AWS)
- Your workspace have to be enabled for Unity Catalog.
For additional steering, consult with the LakeFlow Join documentation.
