Data Engineering & Data Science

Databricks Runtime 14.3 includes a new capability that allows users to access and analyze Structured Streaming 's internal state data: the State Reader... View the full article

Today, we are excited to introduce DBRX, an open, general-purpose LLM created by Databricks. Across a range of standard benchmarks, DBRX sets a... View the full article

Databricks’ mission is to deliver data intelligence to every enterprise by allowing organizations to understand and use their unique data to build their... View the full article

By Steve Sobel - Global Industry Leader; Communications, Media & Entertainment Today Databricks and Adobe are excited to announce a strategic partnership focused... View the full article

As new Generative AI capabilities continue to emerge with heightened customer expectations, data modernization and migration to the cloud have become critical success... View the full article

With the releases of Apache Spark 3.4 and 3.5 in 2023, we focused heavily on improving PySpark performance, flexibility, and ease of use... View the full article

The GGUF file format is a binary file format used for storing and loading model weights for the GGML library. The library documentation... View the full article

In the previous blog , we discussed how to securely access Azure Data Services from Azure Databricks using Virtual Network Service Endpoints or... View the full article

Introduction After a whirlwind year of developments in 2023, many enterprises are eager to adopt increasingly capable generative AI models to supercharge their... View the full article

Next-generation customer experiences are built upon data and insights derived from various touchpoints. Through these, marketers can detect subtle differences in customer needs... View the full article

Today, we are thrilled to announce that Lilac is joining Databricks. Lilac is a scalable, user-friendly tool for data scientists to search, cluster... View the full article

On ecommerce platforms, a good product description can make an item stand out and drive sales. A good product description should not only... View the full article

Game development is a multifaceted journey that stretches from the initial concept to post-launch support and live operations. At the heart of this... View the full article

Artificial Intelligence is top-of-mind with every C-suite in Retail & Consumer Goods. Companies see the potential to deliver better customer service, derive faster... View the full article

Today, we are excited to announce the general availability of Feature Serving. Features play a pivotal role in AI Applications, typically requiring considerable... View the full article

"Building vehicles that are more like smartphones is the future. We're about to change the ride just like Apple and all the smartphone... View the full article

This article explains the concept of regularization and its significance in machine learning and deep learning. We have discussed how regularization can be used to enhance the performance of linear models, as well as how it can be applied to improve the performance of deep learning models.View the full article

This post was written in collaboration with Jason Labonte, Chief Executive Officer, Veritas Data Research In the realm of healthcare and life sciences... View the full article

Today, we're excited to announce the launch of Brickbuilder Unity Catalog Accelerators. This is an expansion to the Brickbuilder Accelerator program, which pairs... View the full article

The DataFrame equality test functions were introduced in Apache Spark™ 3.5 and Databricks Runtime 14.2 to simplify PySpark unit testing. The full set o... View the full article

This blog continues our series looking at advancements from 2023 to the serverless data warehouse Databricks SQL . The best data warehouse is... View the full article

KX and Databricks have partnered to develop time series analytics solutions for the capital markets sector to support many use cases including quant... View the full article

Check out our LLM Solution Accelerators for Retail for more details and to download the notebooks. Product recommendations are a core feature of... View the full article

StreamNative, a leading Apache Pulsar-based real-time data platform solutions provider, and Databricks, the Data Intelligence Platform, are thrilled to announce the enhanced Pulsar-Spark... View the full article

We are thrilled to announce major improvements to the search capabilities in your Databricks workspace. These enhancements build on DatabricksIQ, the Data Intelligence... View the full article

Special thanks to Barb MacLean, SVP, Head of Technology Operations and Implementation at Coastal Community Bank (Coastal) and Rob Cavallo, President at Cavallo... View the full article

Artificial Intelligence (AI) is going to be embedded in every product and service a business produces and customers interact with. With Generative AI... View the full article

If you are considering transitioning from Microsoft Windows to another operating system that suits your needs, check out these five Linux distributions for data science and machine learning.View the full article

Want to start your data science journey from home, for free, and work at your own pace? Have a dive into this data science roadmap using the YouTube series.View the full article

With Game Developers Conference a week away, we're thrilled to present the 2nd Edition of Databricks' Ultimate Guide to Game Data and AI... View the full article

This post is the second part of our two-part series on the latest performance improvements of stateful pipelines. The first part of this... View the full article

(This post written in collaboration with Zeqiu (Ellen) Wu and Yushi Hu , both PhD students affiliated with the University of Washington, and... View the full article

This blog was written in collaboration with Tim Sedlak, Senior Solutions Architect at Stardog In healthcare and life sciences, accuracy is everything. That's... View the full article

Introduction On January 4th, a new era in digital marketing began as Google initiated the gradual removal of third-party cookies, marking a seismic... View the full article

Special thanks to Phillip Jones, Senior Product Manager, and Harshal Brahmbhatt, Systems Engineer from Cloudflare for their contributions to this blog. Organizations across... View the full article

In today's environment, proactive cybersecurity is crucial to any public sector agency. For many organizations, log data that security professionals need for effective... View the full article

Today, we are excited to announce that Unity Catalog Volumes is now generally available on AWS, Azure, and GCP. Unity Catalog provides a... View the full article

We are excited to announce the upcoming general availability of Azure Private Link support for Databricks SQL (DBSQL) Serverless, planned in April 2024... View the full article

About UK Power Networks UK Power Networks is the largest electricity distributor in the UK. It maintains electricity cables and lines in London... View the full article

KDnuggets' latest original cheat sheet covers Jupyter Notebook magic methods. Check it out now and become a notebook magician.View the full article

Simple explanations, no matter what your level is right now.View the full article

For the past two years, Databricks has collaborated with leading consulting partners to build innovative solutions for industry, migration, and data and AI... View the full article

In the dynamic realm of AI-driven forecasting, businesses navigate a landscape where strategic choices shape their trajectory. One such pivotal decision was made... View the full article

Pretrained large language models aren’t particularly good at responding in concise, coherent sentences out of the box. At a minimum, they have to b... View the full article

What is the US Air Force (USAF) Hackathon? The Air Force Test Center (AFTC) Data Hackathon is a consortium of test experts across... View the full article

This code based tutorial provides a brief introduction to Sentiment Analysis, a method used to predict emotions, similar to a digital psychologist.View the full article

In April 2023 we announced the release of Databricks ARC to enable simple, automated linking of data within a single table. Today we... View the full article

This blog was written in collaboration with Anand Iyer, PhD, MBA, Chief Analytics Officer and Abhi Kumbara, Data Science Manager at Welldoc The... View the full article

As Chief Scientist (Neural Networks) at Databricks, I lead our research team toward the goal of giving everyone the ability to build and... View the full article

In this blog post, we will share how you can use Databricks SQL Materialized Views with Lakeview dashboards to deliver fresh data and... View the full article