Data Engineering & Data Science

Since its launch in 2023, Databricks Assistant has grown to hundreds of thousands of monthly users, including developers at major enterprises like Rivian... View the full article

Introduction Databricks has joined forces with the Virtue Foundation through Databricks for Good, a grassroots initiative providing pro bono professional services to drive... View the full article

Staying competitive in Major League Soccer (MLS) demands building and maintaining a strong squad through strategic roster planning and smart, effective navigation of... View the full article

Czech savings bank Česká spořitelna , a division of Austria’s Erste Group , recently collaborated with AI solution builder DataSentics to explore the... View the full article

Large language models are improving rapidly; to date, this improvement has largely been measured via academic benchmarks. These benchmarks, such as MMLU and... View the full article

We’re excited to announce the Public Preview of Query Git integration as part of the new SQL Editor . Git support for queries... View the full article

We’re excited to announce a joint effort between Databricks for Games and GameAnalytics. This blog and associated code will help our mutual customers... View the full article

Book at meeting wtih Databricks at NRF 2025! As we approach January 2025, the retail industry is gearing up for another groundbreaking Retail's... View the full article

Seven West Media’s 7plus is one of Australia’s leading streaming platforms for broadcast VOD (video on demand), enabling audiences to livestream broadcast content... View the full article

While nearly 80% of the world’s data is in video format, enabling search and understanding on video data has historically been a challenging... View the full article

We just followed the documentation online, and within a few hours, we were operational and started running a job. We never had any... View the full article

As enterprises build agent systems to deliver high quality AI apps, we continue to deliver optimizations to deliver best overall cost-efficiency for our... View the full article

In this first part of a two-part blog series, we demonstrate how generative AI coupled with customer data can help marketing teams generate... View the full article

We’re excited to announce the Public Preview of Hive Metastore (HMS) and AWS Glue Federation in Unity Catalog! This new capability enables Unity... View the full article

What makes a great partnership? For Databricks and AWS, it’s not just about building together—it’s about helping businesses succeed together. At AWS re:Invent... View the full article

We are pleased to announce the winners of the Databricks Generative AI Startup Challenge , a competition held in collaboration with AWS to... View the full article

Introduction Building production-grade, scalable, and fault tolerant Generative AI solutions requires having reliable LLM availability. Your LLM endpoints must be ready to meet... View the full article

Inspiration Going on vacation is an enjoyable experience, but planning the trip can take time and effort for most people. There are numerous... View the full article

Data engineering teams are frequently tasked with building bespoke ingestion solutions for myriad custom, proprietary, or industry-specific data sources. Many teams find that... View the full article

* Explore how startups using Databricks achieve higher revenue and innovation. * Learn about the Databricks Unicorn Index and its insights. * Discover real-world success stories from unicorns and emerging unicorns powered by the Databricks Data Intelligence Platform. View the full article

In today’s rapidly evolving technology landscape, generative artificial intelligence (GenAI) is revolutionizing the way organizations work and is opening up new worlds of... View the full article

Iceberg maintains consistency and atomicity of metadata files. Learn how to connect Unity Catalog's Iceberg REST APIs to Snowflake to read a single source data file as Iceberg. View the full article

Databricks is proud to be a platinum sponsor of NeurIPS 2024. The conference runs from December 10 to 15 in Vancouver, British Columbia... View the full article

Our customers continue to shift from monolithic prompts with general-purpose models to specialized agent systems to achieve the quality needed to drive ROI... View the full article

The explosion of data from devices, applications, and systems has driven the need for scalable, efficient storage and analytics solutions. Amazon S3, known for its durability and flexibility, evolves further with S3 Tables, enabling businesses to query and analyze massive datasets directly from storage. This innovation eliminates the complexity of traditional infrastructure while powering advanced […]View the full article

Equiniti wanted to centralize data and insights to its operations. To this end, it utilized the Databricks Data Intelligence Platform and Mosaic AI tools to enhance customer experience and drive innovation. View the full article

Data integration is an integral part of modern business strategy, enabling businesses to convert raw data into actionable information and make data-driven decisions. Tools like Apache Airflow are used and popular for workflow automation. However, its technical complexities and steeper learning curve can create a challenge for teams that require an efficient real-time data pipeline. […]View the full article

Data preparation tools are very important in the analytics process. They transform raw data into a clean and structured format ready for analysis. These tools simplify complex data-wrangling tasks like cleaning, merging, and formatting, thus saving precious time for analysts and data teams. Whether you are a beginner or an experienced professional, the right data […]View the full article

At Databricks, AutoML is our low-code/no-code model training API that empowers customers to create quality machine learning (ML) models with their data on... View the full article

In recent years, artificial intelligence has transformed from an aspirational technology to a driver of manufacturing innovation and efficiency. Understanding both the current... View the full article

Databricks launches two new self-paced trainings to enhance SQL and AI-powered analytics skills The "Get Started with SQL analytics and BI" course covers how to use Databricks SQL for data analysis and Databricks AI/BI Dashboards and Genie spaces Additional courses being developed include "Databricks AI/BI for self-service analytics" and a deep dive for data analysts on building AI/BI Dashboards and Genie Spaces View the full article

In today’s data-driven world, organizations are constantly seeking efficient ways to process and analyze vast amounts of information across data lakes and warehouses. Enter Amazon SageMaker Lakehouse, which you can use to unify all your data across Amazon Simple Storage Service (Amazon S3) data lakes and Amazon Redshift data warehouses, helping you build powerful analytics and AI and machine learning (AI/ML) applications on a single copy of data. SageMaker Lakehouse gives you the flexibility to access and query your data in-place with all Apache Iceberg compatible tools and engines. This opens up exciting possibilities for Open Source Apache Spark users who want to use …

Established in 2020, EVPassport aims to transform the electric vehicle charging experience. Specializing in multi-family residences, hospitality, retail, workplaces, and commercial parking environments... View the full article

The world of artificial intelligence (AI) and data analytics is about to get a significant boost, thanks to Databricks’ collaboration with NVIDIA. This... View the full article

We’re thrilled to announce that Databricks has been recognized as a winner in multiple categories at the 2024 AWS Partner of the Year... View the full article

Introduction Business intelligence (BI) is undergoing a transformation as data intelligence (DI) brings democratized access to data to everyone across organizations. DI refers... View the full article

Predictive Optimization (PO) enhances the performance of Unity Catalog managed tables by intelligently optimizing data layouts, leading to significant improvements in query performance... View the full article

Executive Summary In this blog post we explore how private equity (PE) firms can leverage data intelligence to enhance portfolio returns. We highlight... View the full article

We are excited to announce the Public Preview of Cross-Platform View Sharing. Available today, it allows data providers to share views across different... View the full article

Databricks is a well-known cloud-based data engineering, processing, and analytics platform. One of its key functions is DATEDIFF(date_diff()) used by data professionals widely. The DATEDIFF function in Databricks is very helpful in analyzing time-based data. Using this function helps the user do complex operations like finding time differences between two date values. It is used […]View the full article

Data is everywhere. We make huge amounts of data every day from our social media interactions to the things we buy online. According to expert predictions, data will globally surpass 175 zettabytes by 2025, a figure that is nearly unfathomable. But having data isn’t enough; you need to use it in the right way to […]View the full article

An ETL tool, which has become the critical choice for any organization today, is tied directly to the ever-growing importance of data integration. However, both Matillion and Talend are among the most used ETL tools, providing different functionalities suited to different business needs. Irrespective of whether it is a small business or an enterprise, what […]View the full article

For today’s manufacturers, streamlined and automated workflows are crucial for overcoming challenges such as manual data management and equipment downtime. By leveraging automated... View the full article

Large language models are revolutionizing how we interact with technology by leveraging advanced natural language processing to perform complex tasks. In recent years... View the full article

In today's rapidly changing digital world, consumer data protection and privacy regulations are reshaping how businesses interact with their customers. These changes can... View the full article

The Databricks Serverless compute infrastructure launches and manages millions of virtual machines (VMs) each day across three major cloud providers, and it is... View the full article

Election betting volume reveals challenges for gaming platforms in collecting player data. Learn how RudderStack's warehouse-native CDP scales cost effectively.View the full article

"We are delving deeper into the capabilities of MLFlow tracing. This functionality will be instrumental in diagnosing performance issues and enhancing the quality... View the full article

Introduction Data is power. But in retail banking, it’s about turning that power into actionable insights while carefully navigating data security risks. Financial... View the full article

We are thrilled to announce the winners of the Generative AI World Cup! This event brought together over 1500 data scientists and AI... View the full article