Data Engineering & Data Science

What is AWS Glue AWS Glue is a serverless integration service that provides a simple, faster, and cheaper approach to discovering, preparing, and integrating data for modern ETL(Extract, Transform & Load) pipelines. Hence, data can be Extracted from the source, Transformed the way it is required, and Loaded into the data warehouse. It has a […]View the full article

In today's rapidly evolving technological landscape, the intersection of data and artificial intelligence (AI) has become a critical focus for organizations across industries... View the full article

Rolls-Royce has witnessed the transformative power of the Databricks Data Intelligence Platform in various AI projects. One example is a collaboration between Rolls-Royce... View the full article

Fueled by the exponential growth in external data and AI for innovation, organizations across all industries are looking for effective ways to collaborate... View the full article

With the increase in data size and the diversity of data sources and destinations, companies and data teams are always on the lookout for tools that can simplify creating and managing data workflows. Many of these teams target cloud services because of their simplicity, low cost, and ability to scale and process terabytes of data. […]View the full article

Training a high-quality machine learning model requires careful data and feature preparation. To fully utilize raw data stored as tables in Databricks, running... View the full article

At Data and AI Summit, we announced the general availability of Databricks Lakehouse Monitoring . Our unified approach to monitoring data and AI... View the full article

We’re excited to announce the Public Preview of LakeFlow Connect for SQL Server, Salesforce, and Workday. These ingestion connectors enable simple and efficient... View the full article

Companies across all industries want to share data with each other to enable collaboration and accelerate innovation. However, these organizations often use different... View the full article

We are excited to announce a range of new integrations that will allow our customers to access and derive insights from their data... View the full article

Introduction An organization adopting new technologies or on a modernization journey typically focuses on upcoming tools, their features and potential performance/cost improvements under... View the full article

Financial Valuations & Comparative Analysis Financial institutions specialized in capital markets such as hedge funds, market makers and pension funds have long been... View the full article

The transformative potential of artificial intelligence (AI) is undeniable. From productivity efficiency, to cost savings, and improved decision-making across all industries, AI is... View the full article

Introduction Time series forecasting serves as the foundation for inventory and demand management in most enterprises. Using data from past periods along with... View the full article

Today, we are excited to announce that Lakehouse Federation in Unity Catalog is now Generally Available (GA) across AWS, Azure, and GCP! Lakehouse... View the full article

Dataricks is thrilled to announce the General Availability (GA) of Primary Key (PK) and Foreign Key (FK) constraints, starting in Databricks Runtime 15.2... View the full article

As the Data Platform team at Databricks, we leverage our own platform to provide an intuitive, composable, and comprehensive Data and AI platform... View the full article

The communications industry is experiencing immense change due to rapid technological advancements and evolving market trends. Communications service providers (CSP) build various solutions... View the full article

We are excited to partner with Meta to release the Llama 3.1 series of models on Databricks, further advancing the standard of powerful... View the full article

Evaluating long-form LLM outputs quickly and accurately is critical for rapid AI development. As a result, many developers wish to deploy LLM-as-judge methods... View the full article

Today, we're thrilled to announce that Mosaic AI Model Training's support for fine-tuning GenAI models is now available in Public Preview. At Databricks... View the full article

Written in collaboration with Navin Sharma and Joe Pindell, Stardog Across industries, the impact of post-delivery failure costs (recalls, warranty claims, lost goodwill... View the full article

How long should you train your language model? How large should your model be? In today's generative AI landscape, these are multi-million dollar... View the full article

Forecasting models are critical for many businesses to predict future trends, but their accuracy depends heavily on the quality of the input data... View the full article

Unlocking True Water Risk Assessment Across Insurance, Finance, Public Safety, and Beyond Check out the solution accelerator to download the notebooks referred to... View the full article

We are excited to announce the General Availability of serverless compute for notebooks, jobs and Delta Live Tables (DLT) on AWS and Azure... View the full article

Hallucinations in large language models (LLMs) occur when models produce responses that do not align with factual reality or the provided context. This... View the full article

We are thrilled to welcome the Prodvana team to Databricks. At Databricks, we are building one of the world’s largest multi-cloud platforms to... View the full article

We are proud to announce two new analyst reports recognizing Databricks in the data engineering and data streaming space: IDC MarketScape: Worldwide Analytic... View the full article

Databricks announced the public preview of Mosaic AI Agent Framework & Agent Evaluation alongside our Generative AI Cookbook at the Data + AI... View the full article

Mixture-of-Experts (MoE) has emerged as a promising LLM architecture for efficient training and inference. MoE models like DBRX , which use multiple expert... View the full article

Generative AI (GenAI) can unlock immense value. Organizations are cognizant of the potential but wary of the need to make smart choices about... View the full article

Introduction Financial institutions face a demanding environment with complex regulatory examinations and a pressing need for flexible and comprehensive risk management solutions. The... View the full article

Today, we are thrilled to announce the general availability of Databricks Assistant and AI-Generated Comments on all cloud platforms . Our mission at... View the full article

The recent Data + AI Summit 2024 was our biggest ever. Over 16,000 of our top customers, prospects, and partners attended in person... View the full article

We’re excited to introduce a revamped Catalog Explorer to streamline your day to day interactions, now live across your Unity Catalog-enabled workspaces. The... View the full article

Thousands of data architects, engineers, and scientists met at Data + AI Summit in San Francisco to hear from industry luminaries like Fei... View the full article

Radiology is an important component of diagnosing and treating disease through medical imaging procedures such as X-rays, computed tomography (CT), magnetic resonance imaging... View the full article

Enhancing DLT development experience is a core focus because it directly impacts the efficiency and satisfaction of developers building data pipelines with DLT... View the full article

In the insurance sector, customers demand personalized, fast, and efficient service that addresses their needs. Meanwhile, insurance agents must access a large amount... View the full article

We are excited to announce that Gartner has recognized Databricks as a Leader in the 2024 Gartner® Magic Quadrant™ for Data Science and... View the full article

Today, we are excited to announce Databricks LakeFlow, a new solution that contains everything you need to build and operate production data pipelines... View the full article

At Databricks, our mission is to democratize data + AI. An open approach to sharing and collaboration is critical to maximize reach and... View the full article

We are excited to announce that we are open sourcing Unity Catalog, the industry’s first open source catalog for data and AI governance... View the full article

In an era marked by rapid advancements in artificial intelligence and an explosion of data and Gen AI tools, enterprises face fragmented data... View the full article

The annual Data Team Awards showcase how different data teams from across the globe are delivering solutions to some of the world’s most... View the full article

“FactSet’s mission is to empower clients to make data-driven decisions and supercharge their workflows and productivity. To deliver AI-driven solutions across our entire... View the full article

Today, we are excited to announce Databricks AI/BI , a new type of business intelligence product built from the ground up to deeply... View the full article

Generative AI fever shows no signs of cooling off. As pressure and excitement build to execute strong GenAI strategies, data leaders and practitioners... View the full article

We're excited to announce the General Availability of Databricks Predictive Optimization. This capability intelligently optimizes your table data layouts for faster queries and... View the full article