Data Engineering & Data Science

In today’s data-driven world, organizations are constantly seeking efficient ways to process and analyze vast amounts of information across data lakes and warehouses. Enter Amazon SageMaker Lakehouse, which you can use to unify all your data across Amazon Simple Storage Service (Amazon S3) data lakes and Amazon Redshift data warehouses, helping you build powerful analytics and AI and machine learning (AI/ML) applications on a single copy of data. SageMaker Lakehouse gives you the flexibility to access and query your data in-place with all Apache Iceberg compatible tools and engines. This opens up exciting possibilities for Open Source Apache Spark users who want to use …

Established in 2020, EVPassport aims to transform the electric vehicle charging experience. Specializing in multi-family residences, hospitality, retail, workplaces, and commercial parking environments... View the full article

The world of artificial intelligence (AI) and data analytics is about to get a significant boost, thanks to Databricks’ collaboration with NVIDIA. This... View the full article

We’re thrilled to announce that Databricks has been recognized as a winner in multiple categories at the 2024 AWS Partner of the Year... View the full article

Introduction Business intelligence (BI) is undergoing a transformation as data intelligence (DI) brings democratized access to data to everyone across organizations. DI refers... View the full article

Predictive Optimization (PO) enhances the performance of Unity Catalog managed tables by intelligently optimizing data layouts, leading to significant improvements in query performance... View the full article

Executive Summary In this blog post we explore how private equity (PE) firms can leverage data intelligence to enhance portfolio returns. We highlight... View the full article

We are excited to announce the Public Preview of Cross-Platform View Sharing. Available today, it allows data providers to share views across different... View the full article

An ETL tool, which has become the critical choice for any organization today, is tied directly to the ever-growing importance of data integration. However, both Matillion and Talend are among the most used ETL tools, providing different functionalities suited to different business needs. Irrespective of whether it is a small business or an enterprise, what […]View the full article

For today’s manufacturers, streamlined and automated workflows are crucial for overcoming challenges such as manual data management and equipment downtime. By leveraging automated... View the full article

Large language models are revolutionizing how we interact with technology by leveraging advanced natural language processing to perform complex tasks. In recent years... View the full article

In today's rapidly changing digital world, consumer data protection and privacy regulations are reshaping how businesses interact with their customers. These changes can... View the full article

The Databricks Serverless compute infrastructure launches and manages millions of virtual machines (VMs) each day across three major cloud providers, and it is... View the full article

Election betting volume reveals challenges for gaming platforms in collecting player data. Learn how RudderStack's warehouse-native CDP scales cost effectively.View the full article

"We are delving deeper into the capabilities of MLFlow tracing. This functionality will be instrumental in diagnosing performance issues and enhancing the quality... View the full article

Introduction Data is power. But in retail banking, it’s about turning that power into actionable insights while carefully navigating data security risks. Financial... View the full article

We are thrilled to announce the winners of the Generative AI World Cup! This event brought together over 1500 data scientists and AI... View the full article

In the rapidly evolving landscape of AI, organizations across all industries are eager to harness its transformational power. However, successful AI utilization and... View the full article

While large language models (LLMs) are increasingly adept at solving general tasks, they can often fall short on specific domains that are dissimilar... View the full article

We’re excited to announce that the Databricks Assistant , now fully hosted and managed within Databricks, is available in public preview! This version... View the full article

We are excited to announce that Azure Private Link is now Generally Available (GA) for Databricks serverless and Mosaic AI Model Serving workloads... View the full article

We’re excited to announce a new integration between Databricks Notebooks and AI/BI Dashboards, enabling you to effortlessly transform insights from your notebooks into... View the full article

Effective data governance is crucial for organizations to harness their data assets. Learn how bp uses Databricks Unity Catalog to enhance their data governance framework, highlighting challenges, strategies, and benefits. View the full article

We are thrilled to unveil the finalists for the Databricks Generative AI Startup Challenge , a competition designed to spotlight innovative early-stage startups... View the full article

We are excited to introduce the gated Public Preview of Predictive Optimization for statistics. Announced at the Data + AI Summit, Predictive Optimization... View the full article

While GenAI is the focus today, most enterprises have been working for a decade or longer to make data intelligence a reality within... View the full article

As organizations increasingly leverage the Databricks Data Intelligence Platform for data and AI needs, upgrading to Unity Catalog is a key step in... View the full article

The Data + AI Skills Gap The “skills gap” has been a concern for CEOs and leaders for many years, and the gap... View the full article

Databricks is turning up the heat at AWS re:Invent 2024 , and we’re bringing more than just data and AI solutions to the... View the full article

In an era where data is the lifeblood of medical advancement, the clinical trial industry finds itself at a critical crossroads. The current... View the full article

Providence Health's extensive network spans 50+ hospitals and numerous other facilities across multiple states, presenting many challenges in predicting patient volume and daily... View the full article

When the Generative AI boom first ignited, every enterprise rushed to deploy the technology. For many, that excitement remains. But companies are also... View the full article

Many AI use cases now depend on transforming unstructured inputs into structured data. Developers are increasingly relying on LLMs to extract structured data... View the full article

The Future: From Rules Engines to Instruction-Following AI Agent Systems In sectors such as banking and insurance, rules engines have long played a... View the full article

Whether you’re coming from healthcare, aerospace, manufacturing, government or any other industries the term big data is no foreign concept; however how that... View the full article

Monolithic to Modular The proof of concept (POC) of any new technology often starts with large, monolithic units that are difficult to characterize... View the full article

The most recent wave of artificial intelligence (AI), spearheaded by the advent and mass adoption of large language models (LLM), showed the potential... View the full article

Over the last few years, we've seen tremendous growth and adoption of Databricks SQL , our intelligent data warehouse purpose-built on the Data... View the full article

Learn more about clickstream data here. We define what it is, explore its benefits, and reveal how RudderStack can streamline your clickstream data pipeline.View the full article

Event streaming involves collecting and analyzing data in real time, increasing efficiency and agility. Learn how RudderStack makes event streaming easy here.View the full article

Today, we are excited to announce the general availability of Databricks Assistant Autocomplete on all cloud platforms. Assistant Autocomplete provides personalized AI-powered code... View the full article

With the pace of modern business and the competitive need for more and more data, organizations now correctly ask whether their data management... View the full article

In industries like finance and retail, vast data is leveraged to generate billions in profits. Yet, in healthcare, the struggle to access critical... View the full article

Retrieval Augmented Generation (RAG) is the top use case for Databricks customers who want to customize AI workflows on their own data. The... View the full article

We consistently hear from our customers that one of the headwinds to transitioning Generative AI applications from pilot to production is the accuracy... View the full article

Summary Databricks Apps, a new way to build and deploy internal data and AI applications, is now available in Public Preview on AWS... View the full article

We are announcing the General Availability of Provider Usage Analytics for Databricks Marketplace providers. This feature lets you analyze lead generation and product... View the full article

We are thrilled to announce that embedding for AI/BI Dashboards is now available. Embedding enables you to seamlessly integrate Databricks AI/BI Dashboards into... View the full article

Introduction Applying Large Language Models (LLMs) for code generation is becoming increasingly prevalent, as it helps you code faster and smarter. A primary... View the full article

Many of our customers are shifting from monolithic prompts with general-purpose models to specialized compound AI systems to achieve the quality needed for... View the full article