Data Engineering & Data Science

Did you know that businesses lose more than $3 trillion a year due to inadequate data management, according to an IBM estimate1 from 2016? The need for effective data management methods has never been clearer as companies struggle with an ever-growing volume of data from many sources. This astonishing figure highlights this business need. Data […]View the full article

Retrieval Augmented Generation (RAG) is the top use case for Databricks customers who want to customize AI workflows on their own data. The... View the full article

We consistently hear from our customers that one of the headwinds to transitioning Generative AI applications from pilot to production is the accuracy... View the full article

Summary Databricks Apps, a new way to build and deploy internal data and AI applications, is now available in Public Preview on AWS... View the full article

We are announcing the General Availability of Provider Usage Analytics for Databricks Marketplace providers. This feature lets you analyze lead generation and product... View the full article

Data warehouses have transformed how companies store and manage data. By centralizing data into a single repository, overall data accessibility and quality improve a lot. A data warehouse is not a single tool but a combination of various processes and tools involved in organizing data in a structured format in a central location. Building an […]View the full article

We are thrilled to announce that embedding for AI/BI Dashboards is now available. Embedding enables you to seamlessly integrate Databricks AI/BI Dashboards into... View the full article

Introduction Applying Large Language Models (LLMs) for code generation is becoming increasingly prevalent, as it helps you code faster and smarter. A primary... View the full article

Many of our customers are shifting from monolithic prompts with general-purpose models to specialized compound AI systems to achieve the quality needed for... View the full article

The buzz around compound AI systems is real, and for good reason. Compound AI systems combine the best parts of multiple AI models... View the full article

Introduction Retrieval-augmented generation (RAG) has revolutionized how enterprises harness their unstructured knowledge base using Large Language Models (LLMs), and its potential has far-reaching... View the full article

What is enterprise AI? Enterprise AI combines artificial intelligence, machine learning and natural language processing (NLP) capabilities with business intelligence. Organizations use enterprise... View the full article

The upcoming AVEVA World Conference in Paris (Oct 14-17) promises to be a landmark event for the future of industrial AI, with Databricks... View the full article

In the two decades since the completion of the first draft of the human genome, the landscape of biological research has undergone a... View the full article

AI has quickly moved from an emerging technology to a business imperative as organizations recognize its potential to transform operations and keep them... View the full article

If you are a data-driven business, then you must know how crucial it is to extract meaningful insights from your data. That’s where Reverse ETL comes into play. I’m guessing you might know what ETL (Extract, Transform, Load) is. It is the process of bringing data into your warehouses. But then, what about getting this […]View the full article

Today most organizations are of the opinion that public APIs should be tapped into and useful information extracted there from. The same, however triggers a sound ETL solution to handle the data correctly. This blog REST API ETL Tools will talk about the various tools that will help you fetch data from Public APIs and […]View the full article

Have you ever opened the billing section of a BigQuery account and got a shocking surprise? You are not alone. BigQuery is a powerful tool, but this power does not come for free all the time. It can quickly deplete your budget if you do not practice good cost management. BigQuery is one of the […]View the full article

In the data engineering industry, managing your data is critical for driving business. Data is gathered from various sources in all shapes and forms, and without the right set of tools, it is impossible to use this data for meaning analysis. If you work with a cloud environment, you must have heard of Microsoft Azure. […]View the full article

Databricks is a popular and powerful unified analytics platform. It helps organizations streamline their data engineering, machine learning, and analytics tasks. As data grows and organizations understand the importance of data-driven decision-making, it becomes important to analyze and optimize the costs of data platforms being used carefully. Without careful management, organizations may end up having […]View the full article

The importance of data quality within an organization cannot be overemphasized as it is a critical aspect of running and maintaining an efficient data warehouse. It tells us how well a dataset meets certain criteria for accuracy, completeness, validity, consistency, uniqueness, timeliness and fitness for purpose. High-quality data ensures that organizations make data-driven decisions to […]View the full article

We are excited to partner with Meta to launch the latest models in the Llama 3 series on the Databricks Data Intelligence Platform... View the full article

“So often I’m asked to produce a dashboard but the request isn’t always clear, even after having a conversation with the person. This... View the full article

We are excited to announce that Databricks now supports Amazon EC2 G6 instances powered by NVIDIA L4 Tensor Core GPUs. This addition marks... View the full article

Special thanks to Daniel Benito (CTO, Bitext), Antonio Valderrabanos(CEO, Bitext), Chen Wang (Lead Solution Architect, AI21 Labs), Robbin Jang (Alliance Manager, AI21 Labs)... View the full article

Launch high-ROI personalization projects that drive engagement and conversions without complex engineering.View the full article

Introduction The bin packing problem is a classic optimization challenge that has far-reaching implications for enterprise organizations across industries. At its core, the... View the full article

Batch Processing is a commonly used data integration method to capture data changes in a database. It runs on a schedule to fetch either incremental or a full data extract. However, this method is inefficient when data latency causes significant performance strain on the source systems. A more modern approach to capturing data changes was […]View the full article

As developers and data engineers build complex applications in Snowflake, monitoring performance is essential for ensuring smooth operation and a positive customer experience. Snowflake operations can be tracked using Snowsight, which provides tools for managing costs, tracking query history, monitoring data loading and transformations, and overseeing data governance activities. Recently, Snowflake introduced “Snowflake Trail,” a […]View the full article

We are excited to announce that Mosaic AI Model Training now supports the full context length of 131K tokens when fine-tuning the Meta... View the full article

Today, we're excited to introduce Databricks Assistant Quick Fix , a powerful new feature designed to automatically correct common, single-line errors such as... View the full article

Generative AI technology has been in the headlines for many months now and there are varying opinions on the state of the technology... View the full article

Attribution analytics are complex. Solve the problem at the root and rapidly deliver accurate paid marketing attribution with RudderStack. View the full article

Run RudderStack Data Apps on top of your Customer 360, and you can ship high-ROI data projects in days, not months.View the full article

At Databricks, we know that data is one of your most valuable assets. Our product and security teams work together to deliver an... View the full article

Transformer models, the backbone of modern language AI, rely on the attention mechanism to process context when generating output. During inference, the attention... View the full article

Are you an entrepreneur or startup with a groundbreaking Generative AI use case built on Databricks? Then we have a Challenge for you... View the full article

Run RudderStack Data Apps on top of your Customer 360, and you can ship high-ROI data projects in days, not months.View the full article

Today, we are excited to announce the support for named parameter markers in the SQL editor. This feature allows you to write parameterized... View the full article

Organizations have begun to built data warehouses and lakes to analyze large amounts of data for insights and business reports. Often time they bring data from multiple data silos into their data lake and also have data stored in particular data stores like NoSQL databases to support different use cases. To analyze all of this […]View the full article

We are updating this blog to show developers how to leverage the latest features of Databricks and the advancements in Spark. Most data... View the full article

Data activation is bigger than any one use case. With a customer 360 in your warehouse, you can transform data activation efforts across your business. View the full article

Personal Access Tokens (PATs) are a convenient way to access services like Azure Databricks or Azure DevOps without logging in with your password... View the full article

We are excited to announce that Databricks was named one of the 2024 Fortune Best Workplaces in Technology™ . This award reflects our... View the full article

1GB of data was referred to as big data in 1999. Nowadays, the term is used for petabytes or even exabytes of data (1024 Petabytes), close to trillions of records from billions of people. In this fast-moving landscape, the key to making a difference is picking up the correct data storage solution for your business. […]View the full article

We’re excited to announce that Hevo Data has achieved the prestigious Snowflake Ready Technology Validation certification! This recognition solidifies our commitment to delivering top-notch data integration solutions that seamlessly work with Snowflake, a leading AI Data Cloud. What is the Snowflake Ready Technology Validation Program? The Snowflake Ready Tech Validation Program identifies and acknowledges partners […]View the full article

Data pipelines and workflows have become an inherent part of the advancements in data engineering, machine learning, and DevOps processes. With ever-increasing scales and complexity, the need to orchestrate these workflows efficiently arises. That is where Apache Airflow steps in —an open-source platform designed to programmatically author, schedule, and monitor workflows. In this blog, we […]View the full article

ETL tools have become important in efficiently handling integrated data. In this blog, we will discuss Fivetran vs AWS Glue, two influential ETL tools on the market. This will help you gain a comprehensive understanding of the product’s features, pricing models, and real-world use cases, helping you choose the right solution. Overview of Fivetran G2 […]View the full article

A Data Pipeline is an indispensable part of a data engineering workflow. It enables the extraction, transformation, and storage of data across disparate data sources and ensures that the right data is available at the right time. Python has emerged as a favorite tool for building such pipelines due to its scripting simplicity, extensive libraries, […]View the full article

In today’s competitive era, data is a catalyst fueling businesses to grow faster. As data volumes increase, fetching insights from this data comes with its challenges. Sure, you can use lakes and marts to dump any data, but ultimately, deriving business insights requires structured data with a faster querying experience. This raises the need for […]View the full article