Data Engineering & Data Science

The secret to good AI is great data. As AI adoption soars, the data platform is the most important component of any enterprise's... View the full article

Delta Lake UniForm, now in GA, enables customers to benefit from Delta Lake’s industry-leading price-performance when connecting to tools in the Iceberg ecosystem. View the full article

We are excited to announce a new data type called variant for semi-structured data. Variant provides an order of magnitude performance improvements compared... View the full article

With more and more customer interactions moving into the digital domain, it's increasingly important that organizations develop insights into online customer behaviors. In... View the full article

Special thanks to Caleb Benningfield and Sam Malissa at Amperity for their valuable insights and contributions to this blog. Today, businesses face a... View the full article

We are excited to announce the general availability of Row Filters and Column Masks in Unity Catalog on AWS , Azure and GCP... View the full article

Salesforce and Databricks are excited to announce an expanded strategic partnership that delivers a powerful new integration - Salesforce Bring Your Own Model... View the full article

Whether you are working on a live title, pre/post production, ongoing maintenance, future releases, another version of a game, or a brand new... View the full article

This blog is authored by Michael Ewins, Director of Engineering at Skyscanner At Skyscanner , we're more than just a flight search engine... View the full article

Over the last few years, Large Language Models (LLMs) have been reshaping the field of natural language, thanks to their transformer-based architectures and... View the full article

The annual Data Team Awards celebrate the critical contributions of data teams to various sectors, spotlighting their role in driving progress and positive... View the full article

The annual Data Team Awards showcase the remarkable efforts of top global enterprise data teams committed to tackling some of today's toughest business... View the full article

In today's digital landscape, secure data sharing is critical to operational efficiency and innovation. Databricks and the Linux Foundation developed Delta Sharing as... View the full article

2 examples of how we’re experimenting with practical customer data use cases for LLMs: Making customer success more efficient and unlocking 1:1 personalization.View the full article

The Data Team Awards annually recognize the indispensable roles of enterprise data teams across industries, celebrating their resilience and innovation from around the... View the full article

If you’ve been following the world of industry-grade LLM technology for the last year, you’ve likely observed a plethora of frameworks and tools... View the full article

We’re excited to announce the General Availability of Delta Lake Liquid Clustering in the Databricks Data Intelligence Platform. Liquid Clustering is an innovative... View the full article

Generative AI (GenAI) is moving incredibly fast. So much so, that in less than two years, GenAI has emerged as one of the... View the full article

We're excited to announce native support in Databricks for ingesting XML data . XML is a popular file format for representing complex data... View the full article

In the last year, the Databricks Money Engineering Team has embarked on an exhilarating journey, achieving nearly double our operational efficiency. We are... View the full article

Following the announcement we made around a suite of tools for Retrieval Augmented Generation, today we are thrilled to announce the general availability... View the full article

We’re excited to announce the Databricks AI Fund, showcasing our commitment to supporting a new generation of founders and startups. View the full article

We are excited to introduce Databricks Assistant Autocomplete now in Public Preview. This feature brings the AI-powered assistant to you in real-time, providing... View the full article

We are thrilled to announce an exciting new feature on the Databricks Marketplace that simplifies the process of setting up private exchanges for... View the full article

In the semiconductor industry, research and development tasks, manufacturing processes, and enterprise planning systems produce an array of data artifacts that can be fused to create an intelligent semiconductor enterprise. Through intelligent data use, an intelligent semiconductor enterprise accelerates time to market, increases manufacturing yield, and enhances product reliability. View the full article

With RudderStack Profiles Cohorts and Activations you can bring business teams closer to the data than ever before without comprising control.View the full article

Databricks is pleased to announce we are ranked #2 in the inaugural annual Glassdoor Award List of Best-Led Companies in 2024 ! At... View the full article

RudderStack Profiles enables every data team to power their business with reliable, complete customer profiles. In this blog, we show you how. View the full article

Successfully building GenAI applications means going beyond just leveraging the latest cutting-edge models. It requires the development of compound AI systems that integrate... View the full article

In the fast-paced landscape of data science and engineering, integrating Artificial Intelligence (AI) has become integral for enhancing productivity. We’ve seen many tools... View the full article

You can’t afford not to solve identity resolution – because when you do the value of every customer data initiative goes up, and the complexity goes down.View the full article

We recently introduced DBRX : an open, state-of-the-art, general-purpose LLM. DBRX was trained, fine-tuned, and evaluated using Mosaic AI Training, scaling training to... View the full article

How we reached 79.9% on the Spider dev dataset with Llama3 8B through savvy prompting and fine-tuning on Databricks. View the full article

AWS data engineering involves designing and implementing data solutions on the Amazon Web Services (AWS) platform. For those aspiring to become AWS data engineers, cracking the interview is somehow difficult. Don’t worry, we’re here to help you! In this blog, we present a comprehensive collection of top AWS data engineer interview questions for you. These questions have been carefully selected to cover a wide range of topics and concepts that are relevant to the AWS Data Engineer role. Understanding the concepts behind these questions would help you to successfully go through the interview. If you are planning to become AWS Data Engineer, I would recommend you to pass AWS…

The annual Data Team Awards highlight how diverse enterprise data teams are tackling some of the most prevalent and complex issues facing the... View the full article

You can build a customer 360 using SQL + dbt, but you’ll face significant challenges. Here are the benefits of declarative data modeling for customer 360.View the full article

Last year, we launched foundation model support in Databricks Model Serving to enable enterprises to build secure and custom GenAI apps on a... View the full article

In December, we announced a new suite of tools to get Generative AI applications to production using Retrieval Augmented Generation (RAG). Since then... View the full article

The Data Team Awards celebrates enterprise data teams' essential role in helping businesses across sectors face their most pressing challenges. With more than... View the full article

Introduction Organizations aiming to become AI and data-driven often need to provide their internal teams with high-quality and trusted data products . Building... View the full article

Data, analytics and AI governance is perhaps the most important yet challenging aspect of any data and AI democratization effort. For your data... View the full article

Moving generative AI applications from the proof of concept stage into production requires control, reliability and data governance. Organizations are turning to open... View the full article

In the fast-paced world of sports, where every second and every play can make a difference, the need for advanced analytics and real-time... View the full article

The generative AI revolution is transforming the way that teams work, and Databricks Assistant leverages the best of these advancements. It allows you... View the full article

The Databricks Data Intelligence Platform offers unparalleled flexibility, allowing users to access nearly instant, horizontally scalable compute resources. This ease of creation can... View the full article

The modern data stack is designed to address the difficulties with data collection, storage, and analysis as the volume and complexity of data... View the full article

A good benchmark is one that clearly shows which models are better and which are worse. The Databricks Mosaic Research team is dedicated... View the full article

We are excited to announce that Databricks on AWS GovCloud is now in public preview and that we recently earned our first FedRAMP®... View the full article

We are proud to announce that Forrester has recognized Databricks as a Leader with the highest scores in both current offering and strategy... View the full article

We are thrilled to announce Unity Catalog Lakeguard , which allows you to run Apache Spark™ workloads in SQL, Python, and Scala with... View the full article