Data Engineering & Data Science

Today, Databricks announces support for the ANSI SQL/PSM scripting language! BEGIN DECLARE txt STRING DEFAULT 'Hello'; SET txt = txt || 'Lakehouse'; IF substr(txt, -1,View the full article

We are experiencing an unprecedented pace of technological innovation driven by AI and data. According to the World Economic Forum’s Future of Jobs report, sixView the full article

A data flow diagram outlines system flow. Discover its key components and how to create one.View the full article

Modernize your data warehouse by migrating to Databricks Legacy enterprise data warehouses (EDWs) are becoming a bottleneck for businesses aiming to scale operations and adoptView the full article

As more organizations adopt lakehouse architectures, migrating from legacy data warehouses like Oracle to modern platforms like Databricks has become a common priority. The benefits—betterView the full article

Why migrate from Netezza to Databricks? The limitations of traditional enterprise data warehouse (EDW) appliances like Netezza are becoming increasingly apparent. These systems have tightlyView the full article

The imperative for modernization Traditional database solutions like SQL Server have struggled to keep up with the demands of modern data workloads due to aView the full article

Before making architectural decisions, it’s worth revisiting the broader migration strategy. In our previous post, we introduced Databricks Professional Services’ approach to complex data warehouseView the full article

Atlassian recently partnered with Databricks to power new data sharing capabilities from Atlassian Analytics, using the Delta Sharing protocol. This partnership allows Atlassian to provideView the full article

With more than 4,300 stores across France, Belgium, Portugal, and Poland, Groupement Mousquetaires is one of France’s leading retail groups, with a diverse portfolio thatView the full article

At Databricks, we’re always working hard to make your queries run faster. Still, there are times when it is helpful to look a little deeperView the full article

Introduction Nuclear energy ranks among the world’s most regulated industries. AI and especially generative AI have created enough impact that thought leaders rank it amongView the full article

Fix fragmented pipelines with data standardization. Learn why, when, and how to standardize data. View the full article

We are happy to announce that Python support for Databricks Asset Bundles is now available in Public Preview! Databricks users have long been able toView the full article

While companies today increasingly recognize the potential of custom AI agents, many still struggle to build and scale these applications. An Economist Impact report showsView the full article

SAP Databricks in SAP Business Data Cloud is now generally available. SAP Databricks is a fully managed version of Databricks included natively as a serviceView the full article

Learn why data silos occur, the risks they pose, and how to identify and break them down.View the full article

SAP Databricks in SAP Business Data Cloud is now generally available on AWS. SAP Databricks is a fully managed version of Databricks included natively asView the full article

SQL has been the lingua franca for structured data analysis for multiple decades, and we have done a lot of work in the last fewView the full article

In our highly (inter)connected world, with the growing impact of AI on almost every facet of business, organizations must redefine, cement, and extend not onlyView the full article

Understanding donor behavior is critical to effective nonprofit fundraising. Learn how Masterworks achieved this with RudderStack.View the full article

The Engagement Challenge in Healthcare In an era where data is abundant and customer expectations are higher than ever, creating a personalized, meaningful experience isView the full article

Introduction In a post-App Tracking Transparency (ATT) world advertising has become all the more challenging. Advertising networks have become more opaque and provide fewer knobsView the full article

We are thrilled to announce that the sharing of materialized views and streaming tables is now available in Public Preview. Streaming Tables (STs) continuously ingestView the full article

The Challenge: Fragmented Data and Delayed Decision-Making Energy companies grapple with a pervasive challenge: data silos. These isolated information systems fragment critical data across variousView the full article

Unleashing the Power of Predictive Analytics and LiveOps Satori and Databricks Integration In the dynamic world of game development, data is the ultimate power-up. AtView the full article

Over the past several months, we’ve made DLT pipelines faster, more intelligent, and easier to manage at scale. DLT now delivers a streamlined, high-performance foundationView the full article

We’re excited to share that Databricks Ventures has invested in the Series B round of Omni. Omni is a leading next-generation business intelligence and embedded analytics platform thatView the full article

Liquid Clustering is an innovative data management technique that significantly simplifies your data layout-related decisions. You only have to choose clustering keys based on queryView the full article

Machine learning and AI are extensively used in manufacturing to optimize processes, enhance quality, and reduce costs. Predictive maintenance algorithms analyze sensor data to anticipateView the full article

RudderStack provides an onboarding experience designed to set your team up for success from the start, and the support continues as you embark on your journey. View the full article

Managing high-value equipment deployed across operational sites is a common challenge for construction firms. In response, many original equipment manufacturers are connecting equipment with theView the full article

Introduction Since our last roundup in February, Databricks AI/BI Dashboards and Genie have received even more exciting enhancements, making our native analytical offering more intuitive,View the full article

Learn how to eliminate data bottlenecks with a solid customer data infrastructure.View the full article

Today, we are thrilled to welcome the Fennel team to Databricks. Fennel improves the efficiency and data freshness of feature engineering pipelines for batch, streamingView the full article

In this blog, we have discussed the major roles of AWS Lake Formation that protect data swamps for AWS Certified Data Engineer Associate Certification – DEA-C01. This favors the candidates preparing for DEA C01 to understand the importance of data protection in the cloud. Scroll up to learn more... View the full article

The Amazon EventBridge connector for Apache Kafka Connect is now generally available. This open-source connector streamlines event integration of Kafka environments with dozens of AWS services and partner integrations without writing custom integration code or running multiple connectors for each target. The connector includes built-in support for Kafka schema registries, offloading large event payloads to S3, and IAM role-based authentication, and is available under the Apache 2.0 license in the AWS GitHub organization. Amazon EventBridge is a serverless event router that enables you to create highly scalable event-driven applications by routing events between your ow…

Topic was moved to forum Artificial Intelligence (AI)

Learn how Sharesies overcame significant challenges managing their customer data and how RudderStack became the core of its modern data stack.View the full article

Imagine a future where decisions that once took days or even weeks happen in seconds, managed flawlessly by intelligent systems without human oversight. Perhaps it’sView the full article

Think of your manufacturing operation like an orchestra - every instrument needs to play in perfect harmony to create a masterpiece. But instead of violinsView the full article

Databricks Assistant is a context-aware AI assistant natively available in the Databricks Data Intelligence Platform. It is designed to simplify SQL and data analysis byView the full article

If the last few weeks have made us certain of something, it’s uncertainty. Supply chains are being completely reimagined to meet the demands of aView the full article

We are excited to introduce the Public Preview of OIDC Token Federation for Enhanced Delta Sharing Security a major security and usability enhancement for whenView the full article

Understanding your customers isn't just about knowing who they are—it's about understanding what they do. Clean, accurate event data is fundamental for this. View the full article

Discover why Apache Iceberg is generating so much excitement, especially for streaming data! This accessible lightboard from Tim Berglund demystifies Iceberg, explaining its history and how it functions within modern data architectures. Learn how Confluent's innovative TableFlow simplifies accessing your Apache Kafka topic data as an Iceberg table in your data lake, eliminating the need for cumbersome integrations. If you're looking to streamline data lake querying and streaming data analysis, this is a must-watch…

Topic was moved to forum Artificial Intelligence (AI)

As organizations scale their Amazon Web Services (AWS) infrastructure, they frequently encounter challenges in orchestrating data and analytics workloads across multiple AWS accounts and AWS Regions. While multi-account strategy is essential for organizational separation and governance, it creates complexity in maintaining secure data pipelines and managing fine-grained permissions particularly when different teams manage resources in separate accounts. Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed orchestration service for Apache Airflow that you can use to set up and operate data pipelines in the Amazon Cloud at scale. Apache Airflow is an open …

Data is the fuel for AI, and organizations are racing to leverage enterprise data to build AI agents, intelligent search, and AI-powered analytics for productivity, deeper insights, and a competitive edge. To power their data clouds, tens of thousands of organizations already choose BigQuery and its integrated AI capabilities. This decade requires AI-native, multimodal, and agentic data-to-AI platforms, with BigQuery leading the way as the autonomous data-to-AI platform. Finally, we have a platform that infuses AI, makes unstructured data a first class citizen, accelerates open lakehouses and embeds governance... View the full article

For decades, businesses have wrestled with unlocking the true potential of their data for real-time operations. Bigtable, Google Cloud's pioneering NoSQL database, has been the engine behind massive-scale, low-latency applications that operate at a global scale. It was purpose-built for the challenges faced in real-time applications, and remains a key piece of Google infrastructure, including YouTube and Ads. This week at Google Cloud Next, we announced continuous materialized views, an expansion of Bigtable’ SQL capabilities. Bigtable SQL and continuous materialized views enable users to build fully-managed, real-time application backends using familiar SQL syntax, incl…