Data Engineering

Are you a startup building core, customer-facing B2B products on Databricks? Then we have a Challenge for you! On the heels of our Generative AI...View the full article

Since our launch on Google Cloud Platform (GCP) in 2021, Databricks on Google Cloud has provided more than 1,500 joint customers with a tightly integrated...View the full article

We’re excited to announce the General Availability of Lakeflow Connect for Salesforce and Workday. Lakeflow Connect introduces no-code ingestion connectors for popular SaaS applications, databases,...View the full article

Introduction Game developers have always looked to build ongoing relationships with its players to maximize the play they bring to the world, and the success...View the full article

As more and more organizations embrace analytics, a wider range of problems are being brought forward to be solved. While data science teams are often...View the full article

We’re excited to announce the Public Preview of the Microsoft Power BI task type in Databricks Workflows, available on Azure, AWS, and GCP. With this...View the full article

Within the big data and analytics space there are two names at the forefront of conversation: Apache Spark and Databricks. While they’re closely related, they serve very different purposes in the data ecosystem. Understanding their core differences is critical for architects, developers, and data engineers looking to build scalable, high-performance data solutions in the cloud. […] The article Databricks vs Apache Spark: Key Differences and When to Use Each was originally published on Build5Nines. To stay up-to-date, Subscribe to the Build5Nines Newsletter. View the full article

Willis Towers Watson (WTW) is a multinational company that provides a wide range of services in commercial insurance brokerage, risk management, employee benefits, and actuarial...View the full article

Qwen models, developed by Alibaba, have shown strong performance in both code completion and instruction tasks. In this blog, we’ll show how you can register...View the full article

Databricks enables organizations to securely share data, AI models, and analytics across teams, partners, and platforms without duplication or vendor lock-in. With Delta Sharing, Databricks...View the full article

Databricks introduced last year Databricks Apps, completing its suite of tools that allows users to create and deploy applications directly on the Databricks Platform. With...View the full article

We’re excited to announce that Anthropic Claude 3.7 Sonnet is now natively available in Databricks across AWS, Azure, and GCP. For the first time, youView the full article

Prisma Cloud is the leading Cloud Security platform that provides comprehensive code-to-cloud visibility into your risks and incidents, offering key remediation capabilities to manage andView the full article

Large language models are challenging to adapt to new enterprise tasks. Prompting is error-prone and achieves limited quality gains, while fine-tuning requires large amounts ofView the full article

Databricks Apps provide a robust platform for building and hosting interactive applications. React is great for building modern, dynamic web applications that need to updateView the full article

Driving Sustainable Aluminum Production: How to Calculate the Material Recovery Ratio with GraphFrames Sustainable production has become an imperative in today’s manufacturing market. According toView the full article

We’re excited to announce the General Availability of Explore in Tableau, a new integration that lets you create Tableau Cloud visualizations directly from Unity Catalog...View the full article

We’re making it easier than ever for Databricks customers to run secure, scalable Apache Spark™ workloads on Unity Catalog Compute with Unity Catalog Lakeguard. In...View the full article

Training AI models for real-world applications require vast amounts of labeled data, which can be costly, time-consuming, and difficult to obtain at scale. Synthetic data...View the full article

We’re excited to announce the General Availability of Hive Metastore (HMS) and AWS Glue Federation in Unity Catalog! This new capability enables Unity Catalog to...View the full article

Introduction In this blog, we share the journey of building a Serverless optimized Artifact Registry from the ground up. The main goals are to ensure...View the full article

At Home Trust, we measure success in terms of relationships. Whether we’re working with individuals or businesses, we strive to help them stay “Ready for...View the full article

Generative AI is transforming how organizations interact with their data, and batch LLM processing has quickly become one of Databricks' most popular use cases. Last...View the full article

In today’s dynamic retail environment, staying connected to customer sentiments is more crucial than ever. With shoppers sharing their experiences across countless platforms, retailers are...View the full article

Earlier this week, we announced new agent development capabilities on Databricks. After speaking with hundreds of customers, we've noticed two common challenges to advancing beyond...View the full article

기업들이 전략적 의사 결정을 내릴 때 데이터 기반 인사이트를 적극 활용함에 따라 데이터 인텔리전스 플랫폼의 최신 트렌드는 더욱 정교하고 확장 가능하며 안전한 솔루션으로 발전하는 방향을...View the full article

As part of our Week of AI agents initiative, we’re introducing new capabilities to help enterprises build and govern high-quality AI agents. To that end,...View the full article

As we mentioned in our blog earlier this week, AI agents require enterprise data integration and output governance to achieve production quality. Today we're launching...View the full article

While 85% of global enterprises already use Generative AI (GenAI), organizations face significant challenges scaling these projects beyond the pilot phase. Even the most advanced...View the full article

Marketers have long dreamed of one-on-one customer engagement, but crafting the volume of messages required for personalized engagement at that level has been a major...View the full article

We’re excited to announce the General Availability of Lakehouse Federation for Google BigQuery and the Public Preview for Oracle and Teradata. Now, you can connect,...View the full article

We’re excited to announce the Public Preview of Automatic Liquid Clustering, powered by Predictive Optimization. This feature automatically applies and updates Liquid Clustering columns on...View the full article

Databricks is excited to announce an expansion to our startup offer, providing game studios access to free credits, expert advice and a data and AI...View the full article

Microsoft has announced the official retirement of Azure Data Studio on February 28, 2026. This decision marks a significant shift in Microsoft’s approach to database development tools, as users are encouraged to transition to Visual Studio Code (VS Code) with the appropriate extensions for database management and SQL development. What is Azure Data Studio? Azure […] The article Azure Data Studio Retires February 28, 2026 was originally published on Build5Nines. To stay up-to-date, Subscribe to the Build5Nines Newsletter. View the full article

When we think of use cases like product recommendations, churn predictions, advertising attribution and fraud detection, a common denominator is they all require us to...View the full article

Introduction AI/BI Dashboards and Genie are evolving at a breakneck pace. In this roundup, we’ll highlight the most impactful updates from the past three months...View the full article

Building an end-to-end AI or ML platform often requires multiple technological layers for storage, analytics, business intelligence (BI) tools, and ML models in order to...View the full article

Summary Databricks will be at GDC this year, demonstrating how game teams can de-risk their development and better know and grow their player base like...View the full article

In today's fast-paced business environment, the ability to quickly access and analyze data is crucial for maintaining a competitive edge. As North America's largest book...View the full article

Book at meeting with Databricks at MWC 2025! As we approach Mobile World Congress (MWC) 2025, the telecommunications industry is poised for a transformative leap...View the full article

For utility companies such as Xcel Energy, wildfire mitigation is critical to protecting electrical infrastructure and minimizing the risk of utility-related ignition events. Typical mitigation...View the full article

Introduction Stateful stream processing refers to processing a continuous stream of events in real-time while maintaining state based on the events seen so far. This...View the full article

The Future of Data and AI Belongs to Open and Portable Platforms The promise of AI has never been greater. As organizations race to...View the full article

Databricks SQL continues to evolve with new features and performance improvements designed to make it simpler, faster, and more cost-efficient. Built on the lakehouse architecture...View the full article

Finetuning Embedding Models for Better Retrieval and RAG TL;DR: Finetuning an embedding model on in-domain data can significantly improve vector search and retrieval-augmented generation (RAG)...View the full article

AI has been with us for years, but the extent of its capabilities is only now becoming apparent. The rise of GenAI tools such...View the full article

At Domino's, we're always looking for innovative ways to improve our customer experience and deliver the perfect pizza. Our latest project, aptly named "Voice of...View the full article

At Domino's, we're always looking for innovative ways to improve our customer experience and deliver the perfect pizza. Our latest project, aptly named... View the full article

Introduction VisitBritain is the official website for tourism to the United Kingdom, designed to help visitors plan their trips and get recommendations on top destinations,...View the full article

As we welcome the new year, we're thrilled to announce several new resources for R users on Databricks: a comprehensive developer guide, the release...View the full article

Introduction VisitBritain is the official website for tourism to the United Kingdom, designed to help visitors plan their trips and get recommendations on... View the full article

As we welcome the new year, we're thrilled to announce several new resources for R users on Databricks: a comprehensive developer guide, the... View the full article

Databricks AI/BI is an AI-powered business intelligence solution native to the Databricks Platform that enables natural language queries and AI-generated insights. AI/BI Dashboards offer a...View the full article

Databricks AI/BI is an AI-powered business intelligence solution native to the Databricks Platform that enables natural language queries and AI-generated insights. AI/BI Dashboards... View the full article

As we continue to navigate the complexities of the modern world, it's becoming increasingly clear that data-driven decision making is the key to unlocking success....View the full article

As we continue to navigate the complexities of the modern world, it's becoming increasingly clear that data-driven decision making is the key to unlocking success....View the full article

As we continue to navigate the complexities of the modern world, it's becoming increasingly clear that data-driven decision making is the key to... View the full article

If you are new to Delta Live Tables, prior to reading this blog we recommend reading Getting Started with Delta Live Tables, which explains how...View the full article

If you are new to Delta Live Tables, prior to reading this blog we recommend reading Getting Started with Delta Live Tables... View the full article

Databricks is excited to introduce enhanced streaming observability within Workflows and Delta Live Tables (DLT) pipelines. This feature provides data engineering teams with robust tools...View the full article

Databricks is excited to introduce enhanced streaming observability within Workflows and Delta Live Tables (DLT) pipelines. This feature provides data engineering teams with... View the full article

At AT&T, we are dedicated to providing innovative, reliable wireless carrier services to our 182 million wireless customers every day; we continually strive to improve...View the full article

At AT&T, we are dedicated to providing innovative, reliable wireless carrier services to our 182 million wireless customers every day; we continually strive... View the full article

Today we are announcing a deep partnership with SAP which we think can be game changing for our industry. In short, it is... View the full article

Utilities Of Today Today’s power grid traces its roots back to the late 1800’s when Pearl Street Station first serviced a handful of... View the full article

We are excited to announce the second edition of the Databricks AI Security Framework (DASF 2.0— download now )! Organizations racing to harness... View the full article

Governance, risk and compliance key to reaping AI rewards The AI revolution is underway, and enterprises are keen to explore how the latest AI...View the full article

Governance, risk and compliance key to reaping AI rewards The AI revolution is underway, and enterprises are keen to explore how the latest... View the full article

GreenLight Biosciences participated in the Generative AI World Cup , a six-week hackathon hosted by Databricks to build an image processing agent that... View the full article

FOX Sports has a long history of driving the evolution of broadcast technology, from its high-definition coverage to experiments with virtual reality. Eventually... View the full article

Introducing Serverless Support for AWS Instance Profiles: Uniform Data Access At Databricks, we continuously strive to simplify data access and drive innovation across... View the full article

Registering new products can be a complex and time-consuming process for both suppliers and retailers. Retailers often report issues with incomplete, inaccurate, or... View the full article

We’re thrilled to announce the General Availability (GA) of Databricks Clean Rooms on AWS and Azure, a significant step forward in enabling secure... View the full article

Databricks welcomes BladeBridge, a proven provider of AI-powered migration solutions for enterprise data warehouses. Together, Databricks and BladeBridge will help enterprises accelerate the... View the full article

Dave & Buster’s Entertainment, Inc. owns and operates over 200 venues in North America that offer premier entertainment and dining experiences to guests... View the full article

In the rapidly evolving landscape of data engineering and analytics, speed, scalability, and simplicity are invaluable. Serverless compute addresses these needs by eliminating... View the full article

Opportunities and Obstacles in Developing Reliable Generative AI for Enterprises Generative AI offers transformative benefits in enterprise application development by providing advanced natural... View the full article

Deepseek-R1 is a state-of-the-art open model that, for the first time, introduces the ‘reasoning’ capability to the open source community. In particular, the... View the full article

Businesses rely on data to drive decisions, uncover trends, and stay ahead of the competition. But raw data is often messy, scattered across multiple sources, and difficult to analyze effectively. ETL data modeling offers a structured approach to transform this chaos into meaningful insights. Extract, Transform, and Load (ETL) isn’t just a technical workflow—it’s a […]View the full article

The world is currently data-driven, and most businesses and organizations extract valuable insights from their data to gain a competitive advantage. This is where ETL (Extract, Transform, and Load) and SQL (Structured Query Language processes come into play. In this write-up, you will explore the relationship between ETL and SQL, analyze how SQL is used […]View the full article

Databricks was built as an open and unified platform to handle huge data workloads at a fraction of the cost of other solutions... View the full article

At Zafin , our mission is to help banks modernize their core infrastructure to deliver exceptional, personalized experiences to their customers. To determine... View the full article

In our previous blog , we explored the methodology recommended by our Professional Services teams for executing complex data warehouse migrations to Databricks... View the full article

This blog describes the new change feed and snapshot capabilities in Apache Spark™ Structured Streaming’s State Reader API. The State Reader API enables... View the full article

The average organization generates 2.5 quintillion bytes1 of data daily. Businesses globally prioritize data management due to its exponential growth. How can organizations extract, convert, and load (ETL) meaningful and useable data with so much to process? A robust architecture is crucial to modern data management. This blog will explain how they can assist your […]View the full article

The term ‘lineage’ mainly creates a genealogy or family background or the manner in which people are related across the generations. Data lineage is no different in concept. It gives a chronological account of the extended family of your data, from where it originated to the intermediate transformations it undergoes and where it ends up. […]View the full article

In today’s fast-paced digital landscape, businesses face the daunting challenge of extracting valuable insights from large amounts of data. The ETL (Extract, Transform, Load) pipeline is the backbone of data processing and analysis. Whether you are a seasoned data engineer or a beginner in this data-driven adventure, this blog will help you build a powerful […]View the full article

In today's fast-paced world, utility companies face numerous challenges when it comes to outage response and restoration, especially during severe weather events. The... View the full article

Databricks has been recognized as one of the winners of the annual Glassdoor Employees’ Choice Awards, a list of the Best Places to... View the full article

Electronic products are evolving at lightning speed, driven by an insatiable demand for new consumer devices, energy, transport, robotics, connectivity, data and beyond... View the full article

SELECT 'Hello world!' COLLATE UNICODE, 'Zdravo svete!' COLLATE SR, 'Γειά σου, Κόσμε!' COLLATE EL, 'Здравствуй, мир!' COLLATE RU, '你好, 世界!' COLLATE ZH, 'Bonjour... View the full article

We are excited to announce that egress control for Databricks serverless and Mosaic AI Model Serving workloads is available in Public Preview on... View the full article

Aon plc is a leading global firm providing risk, reinsurance, retirement, and health solutions. Focusing on data-driven insights, Aon operates in over 120... View the full article

At Databricks, our automation vision is to automate all aspects of the business, making it better, faster, and cheaper. For the sales teams... View the full article

Introduction MLOps is an ongoing journey, not a once-and-done project. It involves a set of practices and organizational behaviors, not just individual tools... View the full article

Every organization is challenged with correctly prioritizing new vulnerabilities that affect a large set of third-party libraries used within their organization. The sheer... View the full article

Javier Lagares is a Principal Data Engineer at HP, where he leads the development of data-driven solutions for the 3D printing business. With... View the full article

AI remains at the forefront of every business leader’s plans for 2025. Overall, 70% of businesses continue to believe AI is critical to... View the full article

We are excited to announce that Gartner has recognized Databricks as a Leader for a fourth consecutive year in the 2024 Gartner® Magic... View the full article

We're excited to announce the Public Preview of credential vending for Unity Catalog’s open APIs, allowing external clients to securely access Unity Catalog... View the full article