We’re excited to announce the upcoming basic availability of Azure Non-public Hyperlink assist for Databricks SQL (DBSQL) Serverless, deliberate in April 2024, with no further prices to be used. We’re additionally thrilled to announce that Azure Storage firewall assist with steady VNet subnet IDs is now typically obtainable for DBSQL Serverless. This weblog will give an outline of the 2 options and related finest practices for securely accessing knowledge in your Azure Storage account from Databricks serverless.
Maximize efficiency and safe workspaces utilizing Azure Databricks serverless community connectivity options
The Databricks Information Intelligence Platform gives strong safety by way of sturdy multi-layered isolation and built-in finest practices, as detailed in our Belief Heart, whereas persevering with to leverage knowledge saved in your present Azure Storage accounts. We construct on this basis and provide two choices to attach your DBSQL Serverless workloads to your Azure Storage accounts securely:
- Configure Azure Storage firewall to permit entry based mostly on steady VNet subnet IDs
- Configure Non-public Endpoints to make use of Non-public Hyperlink to your Storage account.
The diagram beneath exhibits the high-level connections into and out of your Azure Databricks account for serverless. On this weblog, we’ll concentrate on securing your connection between DBSQL Serverless workloads and your Azure Storage.
Azure Non-public Hyperlink for serverless will quickly grow to be GA and is included at no further value
Like many shoppers, you’ll have compliance or governance necessities to maintain sources accessible in your digital community site visitors by way of non-public endpoints. For such eventualities, now you can create and keep non-public endpoints to your Storage accounts and grant entry to these non-public endpoints from serverless workloads in specified Workspaces.
As a part of our upcoming basic availability of Non-public Hyperlink on Azure Databricks for serverless, we’re excited to announce that Non-public Hyperlink connections from Databricks SQL Serverless workloads might be obtainable at no further cost to you! In consequence, your TCO for DBSQL Serverless on Azure Databricks will get an enormous increase. It additionally implies that Non-public Hyperlink connections will carry no further cost as we add assist for added Azure Databricks serverless merchandise and Azure useful resource varieties.
“Azure Databricks’ superior networking options provide safety and ease in managing serverless knowledge transformations and analytics at scale.”
— Jonas Kardell, Information Science Lead, SJ AB
Azure Storage firewall assist with steady VNet subnet IDs
For these not wanting to make use of Non-public Hyperlink, you probably nonetheless have a requirement to lock down entry to your knowledge in Azure Storage accounts to solely licensed workloads operating on licensed networks. Azure Storage firewall lets you limit entry to solely purchasers that entry your Storage account from licensed VNet subnet IDs. With this GA launch, you possibly can configure Databricks to make use of a steady listing of subnets inside our Azure VNets to succeed in out to your Storage. You’ll be able to get hold of this listing of subnet IDs straight within the product and handle entry by including them to your Azure Storage firewall guidelines. Combining this function with Unity Catalog gives layered safety to make sure that solely licensed workloads that even have entry to the fitting Managed Identification can entry knowledge in your Storage.
Handle serverless community connectivity simply throughout quite a few Workspaces
With the Community Connectivity Configuration (NCC), you possibly can simply and centrally handle community connectivity. Utilizing NCC permits mapping connectivity configurations to a number of Workspaces, simplifying administration by lowering the variety of non-public endpoints you must handle. As we proceed to broaden our serverless choices, the NCC will proceed to be the only level of managing connectivity throughout all our serverless merchandise.
Getting Began with Serverless Community Connectivity on Azure Databricks
Azure Storage firewall assist and Azure Non-public Hyperlink can be found on the Premium Tier model of Azure Databricks. Check with our documentation for step-by-step directions on configuring NCC and Azure Storage firewall assist to your Databricks workspaces. Whereas Azure Non-public Hyperlink is in gated public preview, contact your Azure Databricks account group for extra info on the way to enroll. We’re planning to make Azure Non-public Hyperlink assist for Azure Databricks serverless typically obtainable in April 2024.
Please go to our Safety and Belief Heart for extra details about Databricks’ safety finest practices and options obtainable to clients.
