Data Engineering

Since its launch in 2023, Databricks Assistant has grown to hundreds of thousands of monthly users, including developers at major enterprises like Rivian... View the full article

Introduction Databricks has joined forces with the Virtue Foundation through Databricks for Good, a grassroots initiative providing pro bono professional services to drive... View the full article

Staying competitive in Major League Soccer (MLS) demands building and maintaining a strong squad through strategic roster planning and smart, effective navigation of... View the full article

Czech savings bank Česká spořitelna , a division of Austria’s Erste Group , recently collaborated with AI solution builder DataSentics to explore the... View the full article

Large language models are improving rapidly; to date, this improvement has largely been measured via academic benchmarks. These benchmarks, such as MMLU and... View the full article

We’re excited to announce the Public Preview of Query Git integration as part of the new SQL Editor . Git support for queries... View the full article

We’re excited to announce a joint effort between Databricks for Games and GameAnalytics. This blog and associated code will help our mutual customers... View the full article

Book at meeting wtih Databricks at NRF 2025! As we approach January 2025, the retail industry is gearing up for another groundbreaking Retail's... View the full article

Seven West Media’s 7plus is one of Australia’s leading streaming platforms for broadcast VOD (video on demand), enabling audiences to livestream broadcast content... View the full article

While nearly 80% of the world’s data is in video format, enabling search and understanding on video data has historically been a challenging... View the full article

We just followed the documentation online, and within a few hours, we were operational and started running a job. We never had any... View the full article

As enterprises build agent systems to deliver high quality AI apps, we continue to deliver optimizations to deliver best overall cost-efficiency for our... View the full article

In this first part of a two-part blog series, we demonstrate how generative AI coupled with customer data can help marketing teams generate... View the full article

We’re excited to announce the Public Preview of Hive Metastore (HMS) and AWS Glue Federation in Unity Catalog! This new capability enables Unity... View the full article

What makes a great partnership? For Databricks and AWS, it’s not just about building together—it’s about helping businesses succeed together. At AWS re:Invent... View the full article

We are pleased to announce the winners of the Databricks Generative AI Startup Challenge , a competition held in collaboration with AWS to... View the full article

Introduction Building production-grade, scalable, and fault tolerant Generative AI solutions requires having reliable LLM availability. Your LLM endpoints must be ready to meet... View the full article

Inspiration Going on vacation is an enjoyable experience, but planning the trip can take time and effort for most people. There are numerous... View the full article

Data engineering teams are frequently tasked with building bespoke ingestion solutions for myriad custom, proprietary, or industry-specific data sources. Many teams find that... View the full article

* Explore how startups using Databricks achieve higher revenue and innovation. * Learn about the Databricks Unicorn Index and its insights. * Discover real-world success stories from unicorns and emerging unicorns powered by the Databricks Data Intelligence Platform. View the full article

In today’s rapidly evolving technology landscape, generative artificial intelligence (GenAI) is revolutionizing the way organizations work and is opening up new worlds of... View the full article

Iceberg maintains consistency and atomicity of metadata files. Learn how to connect Unity Catalog's Iceberg REST APIs to Snowflake to read a single source data file as Iceberg. View the full article

Databricks is proud to be a platinum sponsor of NeurIPS 2024. The conference runs from December 10 to 15 in Vancouver, British Columbia... View the full article

Our customers continue to shift from monolithic prompts with general-purpose models to specialized agent systems to achieve the quality needed to drive ROI... View the full article

Equiniti wanted to centralize data and insights to its operations. To this end, it utilized the Databricks Data Intelligence Platform and Mosaic AI tools to enhance customer experience and drive innovation. View the full article

At Databricks, AutoML is our low-code/no-code model training API that empowers customers to create quality machine learning (ML) models with their data on... View the full article

In recent years, artificial intelligence has transformed from an aspirational technology to a driver of manufacturing innovation and efficiency. Understanding both the current... View the full article

Databricks launches two new self-paced trainings to enhance SQL and AI-powered analytics skills The "Get Started with SQL analytics and BI" course covers how to use Databricks SQL for data analysis and Databricks AI/BI Dashboards and Genie spaces Additional courses being developed include "Databricks AI/BI for self-service analytics" and a deep dive for data analysts on building AI/BI Dashboards and Genie Spaces View the full article

Established in 2020, EVPassport aims to transform the electric vehicle charging experience. Specializing in multi-family residences, hospitality, retail, workplaces, and commercial parking environments... View the full article

The world of artificial intelligence (AI) and data analytics is about to get a significant boost, thanks to Databricks’ collaboration with NVIDIA. This... View the full article

We’re thrilled to announce that Databricks has been recognized as a winner in multiple categories at the 2024 AWS Partner of the Year... View the full article

Introduction Business intelligence (BI) is undergoing a transformation as data intelligence (DI) brings democratized access to data to everyone across organizations. DI refers... View the full article

Predictive Optimization (PO) enhances the performance of Unity Catalog managed tables by intelligently optimizing data layouts, leading to significant improvements in query performance... View the full article

Executive Summary In this blog post we explore how private equity (PE) firms can leverage data intelligence to enhance portfolio returns. We highlight... View the full article

We are excited to announce the Public Preview of Cross-Platform View Sharing. Available today, it allows data providers to share views across different... View the full article

An ETL tool, which has become the critical choice for any organization today, is tied directly to the ever-growing importance of data integration. However, both Matillion and Talend are among the most used ETL tools, providing different functionalities suited to different business needs. Irrespective of whether it is a small business or an enterprise, what […]View the full article

For today’s manufacturers, streamlined and automated workflows are crucial for overcoming challenges such as manual data management and equipment downtime. By leveraging automated... View the full article

Large language models are revolutionizing how we interact with technology by leveraging advanced natural language processing to perform complex tasks. In recent years... View the full article

In today's rapidly changing digital world, consumer data protection and privacy regulations are reshaping how businesses interact with their customers. These changes can... View the full article

The Databricks Serverless compute infrastructure launches and manages millions of virtual machines (VMs) each day across three major cloud providers, and it is... View the full article

"We are delving deeper into the capabilities of MLFlow tracing. This functionality will be instrumental in diagnosing performance issues and enhancing the quality... View the full article

Introduction Data is power. But in retail banking, it’s about turning that power into actionable insights while carefully navigating data security risks. Financial... View the full article

We are thrilled to announce the winners of the Generative AI World Cup! This event brought together over 1500 data scientists and AI... View the full article

In the rapidly evolving landscape of AI, organizations across all industries are eager to harness its transformational power. However, successful AI utilization and... View the full article

While large language models (LLMs) are increasingly adept at solving general tasks, they can often fall short on specific domains that are dissimilar... View the full article

We’re excited to announce that the Databricks Assistant , now fully hosted and managed within Databricks, is available in public preview! This version... View the full article

We are excited to announce that Azure Private Link is now Generally Available (GA) for Databricks serverless and Mosaic AI Model Serving workloads... View the full article

We’re excited to announce a new integration between Databricks Notebooks and AI/BI Dashboards, enabling you to effortlessly transform insights from your notebooks into... View the full article

Effective data governance is crucial for organizations to harness their data assets. Learn how bp uses Databricks Unity Catalog to enhance their data governance framework, highlighting challenges, strategies, and benefits. View the full article

We are thrilled to unveil the finalists for the Databricks Generative AI Startup Challenge , a competition designed to spotlight innovative early-stage startups... View the full article

We are excited to introduce the gated Public Preview of Predictive Optimization for statistics. Announced at the Data + AI Summit, Predictive Optimization... View the full article

While GenAI is the focus today, most enterprises have been working for a decade or longer to make data intelligence a reality within... View the full article

As organizations increasingly leverage the Databricks Data Intelligence Platform for data and AI needs, upgrading to Unity Catalog is a key step in... View the full article

The Data + AI Skills Gap The “skills gap” has been a concern for CEOs and leaders for many years, and the gap... View the full article

Databricks is turning up the heat at AWS re:Invent 2024 , and we’re bringing more than just data and AI solutions to the... View the full article

In an era where data is the lifeblood of medical advancement, the clinical trial industry finds itself at a critical crossroads. The current... View the full article

Providence Health's extensive network spans 50+ hospitals and numerous other facilities across multiple states, presenting many challenges in predicting patient volume and daily... View the full article

When the Generative AI boom first ignited, every enterprise rushed to deploy the technology. For many, that excitement remains. But companies are also... View the full article

Many AI use cases now depend on transforming unstructured inputs into structured data. Developers are increasingly relying on LLMs to extract structured data... View the full article

The Future: From Rules Engines to Instruction-Following AI Agent Systems In sectors such as banking and insurance, rules engines have long played a... View the full article

Whether you’re coming from healthcare, aerospace, manufacturing, government or any other industries the term big data is no foreign concept; however how that... View the full article

Monolithic to Modular The proof of concept (POC) of any new technology often starts with large, monolithic units that are difficult to characterize... View the full article

The most recent wave of artificial intelligence (AI), spearheaded by the advent and mass adoption of large language models (LLM), showed the potential... View the full article

Over the last few years, we've seen tremendous growth and adoption of Databricks SQL , our intelligent data warehouse purpose-built on the Data... View the full article

Today, we are excited to announce the general availability of Databricks Assistant Autocomplete on all cloud platforms. Assistant Autocomplete provides personalized AI-powered code... View the full article

With the pace of modern business and the competitive need for more and more data, organizations now correctly ask whether their data management... View the full article

In industries like finance and retail, vast data is leveraged to generate billions in profits. Yet, in healthcare, the struggle to access critical... View the full article

Retrieval Augmented Generation (RAG) is the top use case for Databricks customers who want to customize AI workflows on their own data. The... View the full article

We consistently hear from our customers that one of the headwinds to transitioning Generative AI applications from pilot to production is the accuracy... View the full article

Summary Databricks Apps, a new way to build and deploy internal data and AI applications, is now available in Public Preview on AWS... View the full article

We are announcing the General Availability of Provider Usage Analytics for Databricks Marketplace providers. This feature lets you analyze lead generation and product... View the full article

We are thrilled to announce that embedding for AI/BI Dashboards is now available. Embedding enables you to seamlessly integrate Databricks AI/BI Dashboards into... View the full article

Introduction Applying Large Language Models (LLMs) for code generation is becoming increasingly prevalent, as it helps you code faster and smarter. A primary... View the full article

Many of our customers are shifting from monolithic prompts with general-purpose models to specialized compound AI systems to achieve the quality needed for... View the full article

The buzz around compound AI systems is real, and for good reason. Compound AI systems combine the best parts of multiple AI models... View the full article

Introduction Retrieval-augmented generation (RAG) has revolutionized how enterprises harness their unstructured knowledge base using Large Language Models (LLMs), and its potential has far-reaching... View the full article

What is enterprise AI? Enterprise AI combines artificial intelligence, machine learning and natural language processing (NLP) capabilities with business intelligence. Organizations use enterprise... View the full article

The upcoming AVEVA World Conference in Paris (Oct 14-17) promises to be a landmark event for the future of industrial AI, with Databricks... View the full article

In the two decades since the completion of the first draft of the human genome, the landscape of biological research has undergone a... View the full article

AI has quickly moved from an emerging technology to a business imperative as organizations recognize its potential to transform operations and keep them... View the full article

We are excited to partner with Meta to launch the latest models in the Llama 3 series on the Databricks Data Intelligence Platform... View the full article

“So often I’m asked to produce a dashboard but the request isn’t always clear, even after having a conversation with the person. This... View the full article

We are excited to announce that Databricks now supports Amazon EC2 G6 instances powered by NVIDIA L4 Tensor Core GPUs. This addition marks... View the full article

Special thanks to Daniel Benito (CTO, Bitext), Antonio Valderrabanos(CEO, Bitext), Chen Wang (Lead Solution Architect, AI21 Labs), Robbin Jang (Alliance Manager, AI21 Labs)... View the full article

Introduction The bin packing problem is a classic optimization challenge that has far-reaching implications for enterprise organizations across industries. At its core, the... View the full article

We are excited to announce that Mosaic AI Model Training now supports the full context length of 131K tokens when fine-tuning the Meta... View the full article

Today, we're excited to introduce Databricks Assistant Quick Fix , a powerful new feature designed to automatically correct common, single-line errors such as... View the full article

Generative AI technology has been in the headlines for many months now and there are varying opinions on the state of the technology... View the full article

At Databricks, we know that data is one of your most valuable assets. Our product and security teams work together to deliver an... View the full article

Transformer models, the backbone of modern language AI, rely on the attention mechanism to process context when generating output. During inference, the attention... View the full article

Are you an entrepreneur or startup with a groundbreaking Generative AI use case built on Databricks? Then we have a Challenge for you... View the full article

Today, we are excited to announce the support for named parameter markers in the SQL editor. This feature allows you to write parameterized... View the full article

We are updating this blog to show developers how to leverage the latest features of Databricks and the advancements in Spark. Most data... View the full article

Personal Access Tokens (PATs) are a convenient way to access services like Azure Databricks or Azure DevOps without logging in with your password... View the full article

We are excited to announce that Databricks was named one of the 2024 Fortune Best Workplaces in Technology™ . This award reflects our... View the full article

We are excited to introduce several powerful new capabilities to Mosaic AI Gateway, designed to help our customers accelerate their AI initiatives with... View the full article

Imagine giving your business an intelligent bot to talk to customers. Chatbots are commonly used to talk to customers and provide them with... View the full article

Personalization and scale have historically been mutually exclusive. For all the talk of one-to-one marketing and hyper-personalization , the reality has been that... View the full article

As recently announced at this year’s Data and AI Summit, Databricks AI/BI democratizes business intelligence and analytics across your organization with highly visual... View the full article

Over the past three months, I had the opportunity to work as a Product Management Intern on the Ingestion team at Databricks. During... View the full article