100 new options. 20 new integrations. 2 huge product launches. A primary of its variety buyer convention. At Atlan, 2023 was the yr of constructing the longer term, with knowledge groups which can be envisioning it. And we couldn’t be extra excited to recap the yr, so let’s get began.
The First AI Copilot for Information Groups
Generative AI modified the expertise panorama in 2023 and each knowledge group began fascinated by the methods wherein they may use AI to change into extra productive.
Armed with learnings from once we had been a knowledge group, we seized the chance to make the lives of people of information higher with Generative AI. In April 2023, we referred to as for a company-wide AI hackathon, sourcing concepts for the way forward for knowledge catalogs and knowledge governance.
Two months later, in June 2023, we launched Atlan AI. By partnering with Microsoft and utilizing Azure OpenAI Service, Atlan was the primary to deliver AI superpowers to knowledge catalogs, main a wave of AI innovation for knowledge governance.
Make your group love documentation with Atlan AI
Within the outdated world, everybody uncared for documentation, blocking companies from trusting knowledge and making sooner, higher selections. Atlan AI brings an finish to the ache by auto-generating descriptions for a variety of information belongings in Atlan. All knowledge producers must do is evaluate, edit, and publish.
After 6 months of testing with clients below a closed preview, we’ve seen Atlan AI’s description solutions shine with an acceptance charge of 60%, giving people of information extra time to work on business-critical tasks.
Clarify lineage transformations with Atlan AI
Lineage simplifies transformations by taking complexity away, exhibiting relationships between knowledge belongings. However when a knowledge analyst desires to dive deeper into how and why a knowledge asset was remodeled, the method of parsing via complicated SQL queries can take hours. However not anymore.
Atlan AI helps knowledge analysts and engineers perceive complicated lineage transformations by explaining what’s happening in pure language.
It’s not an exaggeration to say that knowledge catalogs and knowledge governance are altering eternally, and for the higher, due to AI.
The Way forward for Information Governance
In 2023, we hosted a primary of its variety group convention on fashionable knowledge governance — Re:Govern. Practically a thousand people of information got here collectively to listen to visionary knowledge leaders, from corporations like Nasdaq, Fox, Autodesk, Elastic, and HelloFresh, discuss their methods and playbooks for contemporary knowledge governance.
If there was just one key takeaway from Re:Govern, it might be this: the way forward for knowledge governance will look very totally different from its previous.
Amy Raygada (Swiss Market Group) and Mark Kidwell (Autodesk), shared their visionary approaches to constructing a knowledge mesh. Takashi Ueki (Elastic) make clear constructing automating belief via knowledge contracts. And Mihir Modi (FOX) defined his imaginative and prescient for knowledge merchandise and AI in knowledge governance.
Information groups are abandoning the outdated world of guide, siloed, one-size-fits-all approaches for a brand new world the place automation & AI, collaboration, and adaptability are key to success.
That’s why, in 2023, we began innovating in the direction of a future the place knowledge mesh involves life, AI is embedded in our workflows, and guide, conventional governance is automated. With that, Let’s recap 2023’s knowledge governance updates:
Carry your knowledge mesh to life with Atlan Mesh
Atlan Mesh is the primary ever native knowledge mesh expertise in a knowledge catalog. It’s an expertise that caters to knowledge shoppers and improves their understanding of the info property. Right here’s how:
- Information merchandise as first-class residents: Treating knowledge as a product requires a local answer, not workarounds. Information merchandise can now be created & curated natively in Atlan, inside a brand-new house for simple, context-rich discovery for enterprise customers.
- Devoted areas for federated domains: With Atlan Mesh, every area will get its group’s personal workspace and touchdown web page to deal with curated knowledge merchandise and documentation for knowledge shoppers.
- Enterprise lineage: Historically, lineage has been a technical instrument for knowledge producers to grasp affect and discover root causes. Atlan Mesh introduces enterprise lineage for knowledge shoppers, who need to perceive the provenance, not the technical structure, behind how knowledge merchandise are created to information utilization selections.
- Information contracts: To help the creation of information merchandise and proactively bridge the hole between knowledge producers and shoppers, we launched a brand new imaginative and prescient for knowledge contracts in Atlan.
We’ve already began rolling out Atlan Mesh to the primary few clients, who will likely be part of the Atlan Mesh Advisory Council, and we’re excited to construct the way forward for federated, versatile knowledge governance collectively.
Handle compliance in a single house with Tag Administration
As the fashionable knowledge stack continues to evolve, knowledge groups want to make sure the precise folks have the precise entry to the precise knowledge. This entails figuring out delicate knowledge and defending it with the precise entry controls, whereas serving trusted knowledge to knowledge shoppers.
To unravel this problem, we launched Tag Administration — a brand new method to your knowledge group to handle compliance and safety — and have become one of many first Snowflake knowledge governance companions to allow bi-directional tag sync between Snowflake and Atlan.
Tag Administration allows you to:
- Create tags natively or import tags from instruments like Snowflake into Atlan
- Classify knowledge belongings with tags at scale utilizing Atlan’s Playbooks
- Sync tag updates in Atlan again to knowledge sources like Snowflake
With Atlan’s Tag Administration for Snowflake, our group can have one central house to handle tags. Bi-directional tag sync will empower our knowledge producers to tag belongings the place they work and allow our platform group to handle tags and permissions seamlessly.”
Roi Levoso Fernandez, Information Engineering Supervisor, Taxfix
Perceive affect and optimize prices with Recognition & Utilization metrics
As a knowledge chief, you’re all the time trying to get probably the most out of your knowledge, whereas controlling prices. However you want visibility into who’s really utilizing what knowledge wherein instruments.
In 2023, we launched Recognition & Utilization for 4 connectors — Snowflake, Databricks, Energy BI, and Redshift — giving knowledge groups the flexibility to:
- Uncover probably the most or least used belongings by sorting by reputation
- Perceive reputation in lineage with reputation indicators & pop-ups
- See who’s utilizing knowledge with Prime Customers and Current Customers
- Optimize the info property with In style, Sluggish, and Costly queries
With the launch of Recognition and Utilization for Snowflake, Mistertemp, a frontrunner in recruitment and momentary work primarily based in France, deprecated 50% of unused Snowflake tables and over 60% of their Looker belongings:
Every part downstream modified. We had been capable of see each current connection in Fivetran. We might see what was really used. We saved these, and for the whole lot else, we might disconnect.”
David Milosevic, Head of Information & Analytics, Mistertemp
The Period of Lively Metadata
Lively metadata has all the time been core to Atlan’s platform, and in 2023, we noticed it turning into the middle of information estates throughout industries and companies too.
This yr, we had 20 new knowledge leaders, from companies like Docker, Purple, and Datacamp, be a part of the Lively Metadata Pioneers membership — a visionary group that’s pushing the boundaries of metadata ahead by making energetic metadata a precedence. And to shut out the yr, in November 2023, G2 launched its first Lively Metadata Grid Report, pushed by opinions from actual customers, with Atlan being the one chief within the class.
With that, let’s recap 2023’s energetic metadata developments:
Don’t go breaking my coronary heart dashboards with Metadata CI/CD
Affect evaluation is a tiring, time consuming, and disheartening course of for knowledge engineers. However with out it, one small change might break hundreds of dashboards. What if affect evaluation could possibly be proactive and preventative, as an alternative of reactive and guide? Say howdy to Metadata CI/CD.
With integrations for GitHub and GitLab, Metadata CI/CD routinely surfaces impacted belongings proper within the knowledge producer workflow. This implies knowledge engineers don’t must manually test affect and enterprise customers can belief their dashboards, which break much less usually.
After its launch, Metadata CI/CD helped a knowledge group on Atlan understand that the request for a column title change might affect greater than 1,000 business-critical dashboards.
Atlan has been an important assist. We not must depend on these paperwork, and we’re capable of do affect assessments on the click on of a button.”
Nestor Jarquin, World Information & Analytics Lead, Aliaxis
Bringing metadata to everybody’s favourite instrument: spreadsheets
There’s one knowledge instrument that has stood the check of time: Excel.
In 2023, we launched and upgraded our integrations with Microsoft Excel and Google Sheets to allow new use circumstances:
- Speed up documentation by enriching metadata at scale
By importing knowledge belongings from Atlan into Excel or Sheets, now you can doc descriptions, certificates, homeowners, tags, and bulletins to your column belongings utilizing spreadsheet flexibility and sync the metadata updates to Atlan with a single click on.
- Construct belief and preserve finish customers knowledgeable with affect evaluation
Now you can analyze affect sooner by importing affect evaluation right into a spreadsheet and add bulletins to maintain finish customers knowledgeable.
Carry enterprise and knowledge collectively in Microsoft Groups
To create true company-wide adoption, you’ll want to meet your customers the place they work. That’s why Atlan now integrates with Microsoft Groups. Now you can speed up your knowledge and enterprise tasks with higher, cross-functional collaboration round knowledge.
Atlan’s integration with Microsoft Groups allows you to:
- Share knowledge belongings & ask knowledge questions in a Microsoft Groups channels with out leaving Atlan
- Hyperlink essential, context-rich Microsoft Groups threads to Atlan belongings
- Get notifications & alerts in chosen Microsoft Groups channels
Push the boundaries of metadata with Webhooks and new Python and Java SDKs
Getting ready your knowledge property and group for mission-critical knowledge tasks, like AI fashions, wants a platform strategy to metadata. From event-driven metadata use circumstances like alerting to derived metadata use circumstances like Information as a Product scoring, knowledge groups are making the way forward for metadata come to life with this yr’s extensibility enhancements.
In 2023, we launched the Java and Python SDKs, enabling knowledge groups to construct customized energetic metadata use circumstances like:
- Governance reporting: Measure the success your governance initiatives by automating metadata enrichment experiences.
- Customized connections: Join Atlan to your enterprise homegrown programs to allow end-to-end discovery and lineage.
- Derived metadata: Create customized metadata corresponding to a “Metadata Completeness Rating” or “Information as a Product Rating” by analyzing metadata enrichment.
- Metadata migration: Robotically migrate all of your current metadata out of your legacy knowledge catalog to Atlan.
Together with Python and Java SDKs, we additionally launched help for Webhooks in 2023 — opening up the world of event-driven metadata use circumstances.
Webhooks can help you monitor occasions occurring in Atlan, obtain notifications to a URL of your selection, and take motion instantly. For instance, you possibly can create a webhook to ship notifications to your e-mail tackle or collaboration app, like Slack or Microsoft Groups, when a time period is up to date or an asset is tagged.
The probabilities are really limitless and we’re excited to see the way forward for energetic metadata, constructed not by Atlan, however by knowledge groups all over the world.
A Collaboration to Ship Trusted Information
With each instrument within the fashionable knowledge stack turning into more and more siloed, the people of information have gotten siloed as properly.
Information shoppers, who dwell in BI instruments, don’t have visibility into the upstream pipeline world of information producers. So when issues go improper, they’re usually the final to know. And knowledge producers don’t know the way their code adjustments are breaking downstream dashboards.
This yr, we launched native, out-of-the-box connectors for 18 new instruments, protecting areas like Information High quality, Information Observability, Information Orchestration, and Enterprise Intelligence, to deliver the world of information producers and shoppers nearer collectively. Let’s recap 2023’s key integrations.
All-New Partnerships with Information High quality Instruments
Practically 75% of the time, when issues go improper, enterprise stakeholders are the primary to determine knowledge points. It doesn’t must be this fashion. Information high quality is a basic sign into knowledge belief, however it wants to fulfill enterprise customers the place they work, together with the precise metadata context.
That’s why, in 2023, we launched two new out-of-the-box integrations with leaders within the Information High quality and Observability market: Monte Carlo and Soda.
Monte Carlo x Atlan
With Monte Carlo and Atlan, companies can acquire an up-to-date understanding of their knowledge well being, construct belief in knowledge, and help revolutionary new methods to strategy distributed knowledge infrastructure. The native Monte Carlo integration offers you the flexibility to:
- Uncover Monte Carlo incidents and displays in Atlan with Monte Carlo-specific filters.
- Democratize Monte Carlo’s knowledge high quality alerts wherever enterprise customers work with Atlan’s Chrome extension.
- Speed up root trigger and affect evaluation via Monte Carlo incidents being surfaced in Atlan’s column-level lineage.
With Monte Carlo and Atlan, we are able to catch knowledge incidents early on, and supply everybody with clear visibility into the present standing of information accuracy. This has been essential for the manager group to believe we are able to ship on our promise of dependable, reliable knowledge.“
Michael Weiss, Senior Director of Product Administration (NAM, Information Entry and Analytics), NASDAQ
Soda x Atlan
Atlan and Soda’s native integration offers knowledge groups with an intuitive and complete platform to seek out, belief, and use the precise knowledge. The native Soda integration offers you the flexibility to:
- Uncover Soda knowledge high quality metrics and leads to Atlan.
- Inform knowledge customers of information points earlier than they make selections by highlighting Soda’s test leads to Atlan’s Chrome extension.
- Speed up root trigger and affect evaluation via Soda checks being surfaced in Atlan’s column-level lineage.
BI Connectors for Enterprise Adoption
To drive adoption of your knowledge catalog platform and knowledge governance initiatives, you’ll want to meet enterprise customers the place they work: in BI instruments. In 2023, we launched native, out-of-the-box connectors for six new BI instruments to assist knowledge groups drive company-wide adoption:
- Sigma
- Qlik Sense
- Amazon Quicksight
- MicroStrategy
- Thoughtspot
- Sisense
These BI connectors allow companies to:
- Construct a verified, single supply of fact for BI belongings by enabling discovery and documentation of BI belongings in Atlan.
- Allow proactive affect evaluation of downstream dashboards and use circumstances by connecting BI belongings to upstream warehouse, ELT, and supply belongings with cross-system lineage.
- Construct belief in knowledge by surfacing belief alerts from knowledge high quality, observability, and orchestration instruments.
I’ve had at the very least two conversations the place questions on downstream affect would have taken allocation of a number of sources. Then really getting the work completed would have taken at the very least 4 to 6 weeks, however I managed to take a seat alongside one other architect and resolve that inside half-hour, saying ‘When you’re altering the column title or including an additional column, that is what it’s going to interrupt or affect.”
Karthik Ramani, World Head of Information Structure, Dr. Martens
Bringing ELT & Orchestration Context to the Enterprise
Operational metadata can deliver highly effective context from pipeline processes for each enterprise customers and knowledge groups. In 2023, we invested in constructing native connectors to instruments that would assist reply this query: when a pipeline fails, how do you inform a enterprise consumer and assist a knowledge engineer discover the basis trigger sooner? That’s why we constructed our first event-driven integration with Airflow and OpenLineage.
Airflow x OpenLineage x Atlan
In 2023, Airflow, OpenLineage, and Atlan partnered to construct an ecosystem of belief by making real-time pipeline observability a actuality. Right here’s how:
- Get a complete overview of your Airflow pipelines in Atlan by cataloging and documenting Airflow belongings, corresponding to DAGs and duties.
- Observe and monitor your pipeline in a single house with real-time operational metadata like job run particulars and statuses.
Perceive affect of your Airflow pipelines by visualizing how Airflow DAGs and duties connect with your knowledge belongings with cross-system lineage.
Trying out Atlan? Attempt our product tour.
Occupied with knowledge governance? Chat with our group about your governance initiatives and the way Atlan will help.
