Databricks' annual summit continues to be a vibrant gathering for stakeholders in the data ecosystem. Each year, the company unveils innovative technologies, partnerships, and advancements aimed at simplifying the management of both structured and unstructured data. This year's summit maintained that tradition, with a notable emphasis on artificial intelligence (AI).
During the keynote address, CEO Ali Ghodsi introduced a range of innovations at the intersection of data and AI, reinforcing the company's commitment to optimizing governed datasets on the Databricks Data Intelligence Platform. Key highlights included enhancements to Mosaic AI, an image generation model, and a generative AI solution designed for more efficient data analytics.
Here are the major announcements from this year's summit:
1. Unity Catalog Goes Open-Source
In a strategic move to compete with Snowflake’s Polaris Catalog, Databricks has open-sourced its Unity Catalog under an Apache 2.0 license, complete with OpenAPI specifications. This allows other companies to utilize the architecture and code to create catalogs that support various data formats, including Iceberg and Delta/Hudi. The code was shared live during the keynote, with Snowflake expected to follow suit in the next 90 days.
2. Mosaic AI Upgrades for Production-Grade Systems
Mosaic AI, the toolkit for AI application development, received significant enhancements aimed at helping teams create trusted, production-ready compound AI systems. New features include the Mosaic AI Model Training product, an AI Agent framework, an Evaluation framework, and the AI Tools Catalog and AI Gateway for governance. All offerings, aside from the AI tools, are now available in public preview.
3. New Text-to-Image Model for Enterprises
Databricks introduced the private preview of Shutterstock ImageAI, a generative AI model allowing enterprises to create high-fidelity images tailored for various business applications. This model, pre-trained with Mosaic AI and Shutterstock’s trusted image collection, is accessible via Shutterstock’s image generator and can be fine-tuned through Mosaic AI or integrated through APIs.
4. Databricks AI/BI for Intelligent Analytics
To help enterprises democratize access to analytical insights, Databricks launched Databricks AI/BI, a compound AI system integrated with the Data Intelligence Platform. Utilizing AI agents—Dashboards and Genie—this system interprets business queries to produce natural language answers and visualizations. Each agent focuses on specific functions like planning, SQL generation, and visualization, supported by additional components like response ranking and vector indexing. This offering is available to all Databricks SQL Pro and Serverless customers, with Dashboards now generally available and Genie in public preview.
5. Databricks LakeFlow for Simplified Data Engineering
Databricks also unveiled LakeFlow, a unified experience designed to streamline all facets of data engineering, from ingestion to transformation and orchestration. LakeFlow simplifies the traditionally complex process of building and maintaining data pipelines by automating their deployment, operation, and monitoring, with robust support for CI/CD and quality checks at scale. Though not yet in preview, Databricks has opened a waitlist for early access.
6. Partnerships with Nvidia and Gretel
Lastly, Databricks announced significant partnerships with Nvidia and Gretel. The collaboration with Nvidia aims to incorporate native support for CUDA-accelerated computing into Databricks' next-generation vectorized query engine, Photon, enhancing performance for data warehousing and analytics workloads. The partnership with Gretel designates the company as an ISV technology partner, offering high-quality synthetic datasets for developing and customizing machine learning models on Databricks' platform.
Ranking
