Data Engineering & Data Science

Javier Lagares is a Principal Data Engineer at HP, where he leads the development of data-driven solutions for the 3D printing business. With... View the full article

AI remains at the forefront of every business leader’s plans for 2025. Overall, 70% of businesses continue to believe AI is critical to... View the full article

We are excited to announce that Gartner has recognized Databricks as a Leader for a fourth consecutive year in the 2024 Gartner® Magic... View the full article

We're excited to announce the Public Preview of credential vending for Unity Catalog’s open APIs, allowing external clients to securely access Unity Catalog... View the full article

Since its launch in 2023, Databricks Assistant has grown to hundreds of thousands of monthly users, including developers at major enterprises like Rivian... View the full article

Introduction Databricks has joined forces with the Virtue Foundation through Databricks for Good, a grassroots initiative providing pro bono professional services to drive... View the full article

Staying competitive in Major League Soccer (MLS) demands building and maintaining a strong squad through strategic roster planning and smart, effective navigation of... View the full article

Czech savings bank Česká spořitelna , a division of Austria’s Erste Group , recently collaborated with AI solution builder DataSentics to explore the... View the full article

Large language models are improving rapidly; to date, this improvement has largely been measured via academic benchmarks. These benchmarks, such as MMLU and... View the full article

We’re excited to announce the Public Preview of Query Git integration as part of the new SQL Editor . Git support for queries... View the full article

We’re excited to announce a joint effort between Databricks for Games and GameAnalytics. This blog and associated code will help our mutual customers... View the full article

Book at meeting wtih Databricks at NRF 2025! As we approach January 2025, the retail industry is gearing up for another groundbreaking Retail's... View the full article

Seven West Media’s 7plus is one of Australia’s leading streaming platforms for broadcast VOD (video on demand), enabling audiences to livestream broadcast content... View the full article

While nearly 80% of the world’s data is in video format, enabling search and understanding on video data has historically been a challenging... View the full article

We just followed the documentation online, and within a few hours, we were operational and started running a job. We never had any... View the full article

As enterprises build agent systems to deliver high quality AI apps, we continue to deliver optimizations to deliver best overall cost-efficiency for our... View the full article

In this first part of a two-part blog series, we demonstrate how generative AI coupled with customer data can help marketing teams generate... View the full article

We’re excited to announce the Public Preview of Hive Metastore (HMS) and AWS Glue Federation in Unity Catalog! This new capability enables Unity... View the full article

What makes a great partnership? For Databricks and AWS, it’s not just about building together—it’s about helping businesses succeed together. At AWS re:Invent... View the full article

We are pleased to announce the winners of the Databricks Generative AI Startup Challenge , a competition held in collaboration with AWS to... View the full article

Introduction Building production-grade, scalable, and fault tolerant Generative AI solutions requires having reliable LLM availability. Your LLM endpoints must be ready to meet... View the full article

Inspiration Going on vacation is an enjoyable experience, but planning the trip can take time and effort for most people. There are numerous... View the full article

Data engineering teams are frequently tasked with building bespoke ingestion solutions for myriad custom, proprietary, or industry-specific data sources. Many teams find that... View the full article

* Explore how startups using Databricks achieve higher revenue and innovation. * Learn about the Databricks Unicorn Index and its insights. * Discover real-world success stories from unicorns and emerging unicorns powered by the Databricks Data Intelligence Platform. View the full article

In today’s rapidly evolving technology landscape, generative artificial intelligence (GenAI) is revolutionizing the way organizations work and is opening up new worlds of... View the full article

Iceberg maintains consistency and atomicity of metadata files. Learn how to connect Unity Catalog's Iceberg REST APIs to Snowflake to read a single source data file as Iceberg. View the full article

Databricks is proud to be a platinum sponsor of NeurIPS 2024. The conference runs from December 10 to 15 in Vancouver, British Columbia... View the full article

Our customers continue to shift from monolithic prompts with general-purpose models to specialized agent systems to achieve the quality needed to drive ROI... View the full article

Equiniti wanted to centralize data and insights to its operations. To this end, it utilized the Databricks Data Intelligence Platform and Mosaic AI tools to enhance customer experience and drive innovation. View the full article

At Databricks, AutoML is our low-code/no-code model training API that empowers customers to create quality machine learning (ML) models with their data on... View the full article

In recent years, artificial intelligence has transformed from an aspirational technology to a driver of manufacturing innovation and efficiency. Understanding both the current... View the full article

Databricks launches two new self-paced trainings to enhance SQL and AI-powered analytics skills The "Get Started with SQL analytics and BI" course covers how to use Databricks SQL for data analysis and Databricks AI/BI Dashboards and Genie spaces Additional courses being developed include "Databricks AI/BI for self-service analytics" and a deep dive for data analysts on building AI/BI Dashboards and Genie Spaces View the full article

In today’s data-driven world, organizations are constantly seeking efficient ways to process and analyze vast amounts of information across data lakes and warehouses. Enter Amazon SageMaker Lakehouse, which you can use to unify all your data across Amazon Simple Storage Service (Amazon S3) data lakes and Amazon Redshift data warehouses, helping you build powerful analytics and AI and machine learning (AI/ML) applications on a single copy of data. SageMaker Lakehouse gives you the flexibility to access and query your data in-place with all Apache Iceberg compatible tools and engines. This opens up exciting possibilities for Open Source Apache Spark users who want to use …

Established in 2020, EVPassport aims to transform the electric vehicle charging experience. Specializing in multi-family residences, hospitality, retail, workplaces, and commercial parking environments... View the full article

The world of artificial intelligence (AI) and data analytics is about to get a significant boost, thanks to Databricks’ collaboration with NVIDIA. This... View the full article

We’re thrilled to announce that Databricks has been recognized as a winner in multiple categories at the 2024 AWS Partner of the Year... View the full article

Introduction Business intelligence (BI) is undergoing a transformation as data intelligence (DI) brings democratized access to data to everyone across organizations. DI refers... View the full article

Predictive Optimization (PO) enhances the performance of Unity Catalog managed tables by intelligently optimizing data layouts, leading to significant improvements in query performance... View the full article

Executive Summary In this blog post we explore how private equity (PE) firms can leverage data intelligence to enhance portfolio returns. We highlight... View the full article

We are excited to announce the Public Preview of Cross-Platform View Sharing. Available today, it allows data providers to share views across different... View the full article

An ETL tool, which has become the critical choice for any organization today, is tied directly to the ever-growing importance of data integration. However, both Matillion and Talend are among the most used ETL tools, providing different functionalities suited to different business needs. Irrespective of whether it is a small business or an enterprise, what […]View the full article

For today’s manufacturers, streamlined and automated workflows are crucial for overcoming challenges such as manual data management and equipment downtime. By leveraging automated... View the full article

Large language models are revolutionizing how we interact with technology by leveraging advanced natural language processing to perform complex tasks. In recent years... View the full article

In today's rapidly changing digital world, consumer data protection and privacy regulations are reshaping how businesses interact with their customers. These changes can... View the full article

The Databricks Serverless compute infrastructure launches and manages millions of virtual machines (VMs) each day across three major cloud providers, and it is... View the full article

"We are delving deeper into the capabilities of MLFlow tracing. This functionality will be instrumental in diagnosing performance issues and enhancing the quality... View the full article

Introduction Data is power. But in retail banking, it’s about turning that power into actionable insights while carefully navigating data security risks. Financial... View the full article

We are thrilled to announce the winners of the Generative AI World Cup! This event brought together over 1500 data scientists and AI... View the full article

In the rapidly evolving landscape of AI, organizations across all industries are eager to harness its transformational power. However, successful AI utilization and... View the full article

While large language models (LLMs) are increasingly adept at solving general tasks, they can often fall short on specific domains that are dissimilar... View the full article