Data Engineering & Data Science

Game development is a multifaceted journey that stretches from the initial concept to post-launch support and live operations. At the heart of this... View the full article

Artificial Intelligence is top-of-mind with every C-suite in Retail & Consumer Goods. Companies see the potential to deliver better customer service, derive faster... View the full article

Today, we are excited to announce the general availability of Feature Serving. Features play a pivotal role in AI Applications, typically requiring considerable... View the full article

"Building vehicles that are more like smartphones is the future. We're about to change the ride just like Apple and all the smartphone... View the full article

This article explains the concept of regularization and its significance in machine learning and deep learning. We have discussed how regularization can be used to enhance the performance of linear models, as well as how it can be applied to improve the performance of deep learning models.View the full article

This post was written in collaboration with Jason Labonte, Chief Executive Officer, Veritas Data Research In the realm of healthcare and life sciences... View the full article

Today, we're excited to announce the launch of Brickbuilder Unity Catalog Accelerators. This is an expansion to the Brickbuilder Accelerator program, which pairs... View the full article

The DataFrame equality test functions were introduced in Apache Spark™ 3.5 and Databricks Runtime 14.2 to simplify PySpark unit testing. The full set o... View the full article

This blog continues our series looking at advancements from 2023 to the serverless data warehouse Databricks SQL . The best data warehouse is... View the full article

KX and Databricks have partnered to develop time series analytics solutions for the capital markets sector to support many use cases including quant... View the full article

Check out our LLM Solution Accelerators for Retail for more details and to download the notebooks. Product recommendations are a core feature of... View the full article

StreamNative, a leading Apache Pulsar-based real-time data platform solutions provider, and Databricks, the Data Intelligence Platform, are thrilled to announce the enhanced Pulsar-Spark... View the full article

We are thrilled to announce major improvements to the search capabilities in your Databricks workspace. These enhancements build on DatabricksIQ, the Data Intelligence... View the full article

Special thanks to Barb MacLean, SVP, Head of Technology Operations and Implementation at Coastal Community Bank (Coastal) and Rob Cavallo, President at Cavallo... View the full article

Artificial Intelligence (AI) is going to be embedded in every product and service a business produces and customers interact with. With Generative AI... View the full article

If you are considering transitioning from Microsoft Windows to another operating system that suits your needs, check out these five Linux distributions for data science and machine learning.View the full article

Want to start your data science journey from home, for free, and work at your own pace? Have a dive into this data science roadmap using the YouTube series.View the full article

With Game Developers Conference a week away, we're thrilled to present the 2nd Edition of Databricks' Ultimate Guide to Game Data and AI... View the full article

This post is the second part of our two-part series on the latest performance improvements of stateful pipelines. The first part of this... View the full article

(This post written in collaboration with Zeqiu (Ellen) Wu and Yushi Hu , both PhD students affiliated with the University of Washington, and... View the full article

This blog was written in collaboration with Tim Sedlak, Senior Solutions Architect at Stardog In healthcare and life sciences, accuracy is everything. That's... View the full article

Introduction On January 4th, a new era in digital marketing began as Google initiated the gradual removal of third-party cookies, marking a seismic... View the full article

Special thanks to Phillip Jones, Senior Product Manager, and Harshal Brahmbhatt, Systems Engineer from Cloudflare for their contributions to this blog. Organizations across... View the full article

In today's environment, proactive cybersecurity is crucial to any public sector agency. For many organizations, log data that security professionals need for effective... View the full article

Today, we are excited to announce that Unity Catalog Volumes is now generally available on AWS, Azure, and GCP. Unity Catalog provides a... View the full article

We are excited to announce the upcoming general availability of Azure Private Link support for Databricks SQL (DBSQL) Serverless, planned in April 2024... View the full article

About UK Power Networks UK Power Networks is the largest electricity distributor in the UK. It maintains electricity cables and lines in London... View the full article

KDnuggets' latest original cheat sheet covers Jupyter Notebook magic methods. Check it out now and become a notebook magician.View the full article

Simple explanations, no matter what your level is right now.View the full article

For the past two years, Databricks has collaborated with leading consulting partners to build innovative solutions for industry, migration, and data and AI... View the full article

In the dynamic realm of AI-driven forecasting, businesses navigate a landscape where strategic choices shape their trajectory. One such pivotal decision was made... View the full article

Pretrained large language models aren’t particularly good at responding in concise, coherent sentences out of the box. At a minimum, they have to b... View the full article

What is the US Air Force (USAF) Hackathon? The Air Force Test Center (AFTC) Data Hackathon is a consortium of test experts across... View the full article

This code based tutorial provides a brief introduction to Sentiment Analysis, a method used to predict emotions, similar to a digital psychologist.View the full article

In April 2023 we announced the release of Databricks ARC to enable simple, automated linking of data within a single table. Today we... View the full article

This blog was written in collaboration with Anand Iyer, PhD, MBA, Chief Analytics Officer and Abhi Kumbara, Data Science Manager at Welldoc The... View the full article

As Chief Scientist (Neural Networks) at Databricks, I lead our research team toward the goal of giving everyone the ability to build and... View the full article

In this blog post, we will share how you can use Databricks SQL Materialized Views with Lakeview dashboards to deliver fresh data and... View the full article

There are thousands of datasets available to institutional investors, each dataset promising to unlock significant insights in investment decisioning. Across the thousands of... View the full article

Welcome to the blog series covering product advancements in 2023 for Databricks SQL, the serverless data warehouse from Databricks. This is part 2... View the full article

Quantization is a technique for making machine learning models smaller and faster. We quantize Llama2-70B-Chat, producing an equivalent-quality model that generates 2.2x more... View the full article

At Databricks, we believe that AI will change the way that enterprises interact with their data. That’s why today, we're excited to welcome t... View the full article

The post highlights real-world examples of NLP use cases across industries. It also covers NLP's objectives, challenges, and latest research developments.View the full article

Databricks recently announced the Data Intelligence Platform, a natural evolution of the lakehouse architecture we pioneered. The idea of a Data Intelligence Platform... View the full article

Reliable, accurate and trusted data is the most critical requirement for any data application in an enterprise. As Databricks customers increasingly rely on... View the full article

Today, we are announcing the industry's first Generative AI Engineer learning pathway and certification to help ensure that data and AI practitioners have... View the full article

This post is part of a series. Check out Part 1: The Data + AI Trifecta: People, Process, and Platform In the current... View the full article

This is part 1 of a blog series where we look back at the major areas of progress for Databricks SQL in 2023... View the full article

Since COVID, countless articles have been written about the "Great Resignation", including in-depth analysis by the World Economic Forum. One key thing this... View the full article

My personal take on justifying the existence of Data MeshA senior stakeholder at one my projects mentioned that they wanted to decentralise their data platform architecture and democratise data across the organisation. When I heard the words ‘decentralised data architecture’, I was left utterly confused at first! In my then limited experience as a Data Engineer, I had only come across centralised data architectures and they seemed to be working very well. So, I was left wondering what was it that we wanted to solve using a decentralised data architecture? Or were we creating a new problem that did not ever exist in the first place? .. A Prequel to Data Mesh was originally…