Data Engineering & Data Science

The upcoming AVEVA World Conference in Paris (Oct 14-17) promises to be a landmark event for the future of industrial AI, with Databricks... View the full article

In the two decades since the completion of the first draft of the human genome, the landscape of biological research has undergone a... View the full article

AI has quickly moved from an emerging technology to a business imperative as organizations recognize its potential to transform operations and keep them... View the full article

If you are a data-driven business, then you must know how crucial it is to extract meaningful insights from your data. That’s where Reverse ETL comes into play. I’m guessing you might know what ETL (Extract, Transform, Load) is. It is the process of bringing data into your warehouses. But then, what about getting this […]View the full article

Today most organizations are of the opinion that public APIs should be tapped into and useful information extracted there from. The same, however triggers a sound ETL solution to handle the data correctly. This blog REST API ETL Tools will talk about the various tools that will help you fetch data from Public APIs and […]View the full article

Have you ever opened the billing section of a BigQuery account and got a shocking surprise? You are not alone. BigQuery is a powerful tool, but this power does not come for free all the time. It can quickly deplete your budget if you do not practice good cost management. BigQuery is one of the […]View the full article

In the data engineering industry, managing your data is critical for driving business. Data is gathered from various sources in all shapes and forms, and without the right set of tools, it is impossible to use this data for meaning analysis. If you work with a cloud environment, you must have heard of Microsoft Azure. […]View the full article

Databricks is a popular and powerful unified analytics platform. It helps organizations streamline their data engineering, machine learning, and analytics tasks. As data grows and organizations understand the importance of data-driven decision-making, it becomes important to analyze and optimize the costs of data platforms being used carefully. Without careful management, organizations may end up having […]View the full article

The importance of data quality within an organization cannot be overemphasized as it is a critical aspect of running and maintaining an efficient data warehouse. It tells us how well a dataset meets certain criteria for accuracy, completeness, validity, consistency, uniqueness, timeliness and fitness for purpose. High-quality data ensures that organizations make data-driven decisions to […]View the full article

We are excited to partner with Meta to launch the latest models in the Llama 3 series on the Databricks Data Intelligence Platform... View the full article

“So often I’m asked to produce a dashboard but the request isn’t always clear, even after having a conversation with the person. This... View the full article

We are excited to announce that Databricks now supports Amazon EC2 G6 instances powered by NVIDIA L4 Tensor Core GPUs. This addition marks... View the full article

Special thanks to Daniel Benito (CTO, Bitext), Antonio Valderrabanos(CEO, Bitext), Chen Wang (Lead Solution Architect, AI21 Labs), Robbin Jang (Alliance Manager, AI21 Labs)... View the full article

Launch high-ROI personalization projects that drive engagement and conversions without complex engineering.View the full article

Introduction The bin packing problem is a classic optimization challenge that has far-reaching implications for enterprise organizations across industries. At its core, the... View the full article

Batch Processing is a commonly used data integration method to capture data changes in a database. It runs on a schedule to fetch either incremental or a full data extract. However, this method is inefficient when data latency causes significant performance strain on the source systems. A more modern approach to capturing data changes was […]View the full article

As developers and data engineers build complex applications in Snowflake, monitoring performance is essential for ensuring smooth operation and a positive customer experience. Snowflake operations can be tracked using Snowsight, which provides tools for managing costs, tracking query history, monitoring data loading and transformations, and overseeing data governance activities. Recently, Snowflake introduced “Snowflake Trail,” a […]View the full article

We are excited to announce that Mosaic AI Model Training now supports the full context length of 131K tokens when fine-tuning the Meta... View the full article

Today, we're excited to introduce Databricks Assistant Quick Fix , a powerful new feature designed to automatically correct common, single-line errors such as... View the full article

Generative AI technology has been in the headlines for many months now and there are varying opinions on the state of the technology... View the full article

Attribution analytics are complex. Solve the problem at the root and rapidly deliver accurate paid marketing attribution with RudderStack. View the full article

Run RudderStack Data Apps on top of your Customer 360, and you can ship high-ROI data projects in days, not months.View the full article

At Databricks, we know that data is one of your most valuable assets. Our product and security teams work together to deliver an... View the full article

Transformer models, the backbone of modern language AI, rely on the attention mechanism to process context when generating output. During inference, the attention... View the full article

Are you an entrepreneur or startup with a groundbreaking Generative AI use case built on Databricks? Then we have a Challenge for you... View the full article

Run RudderStack Data Apps on top of your Customer 360, and you can ship high-ROI data projects in days, not months.View the full article

Today, we are excited to announce the support for named parameter markers in the SQL editor. This feature allows you to write parameterized... View the full article

Organizations have begun to built data warehouses and lakes to analyze large amounts of data for insights and business reports. Often time they bring data from multiple data silos into their data lake and also have data stored in particular data stores like NoSQL databases to support different use cases. To analyze all of this […]View the full article

We are updating this blog to show developers how to leverage the latest features of Databricks and the advancements in Spark. Most data... View the full article

Data activation is bigger than any one use case. With a customer 360 in your warehouse, you can transform data activation efforts across your business. View the full article

Personal Access Tokens (PATs) are a convenient way to access services like Azure Databricks or Azure DevOps without logging in with your password... View the full article

We are excited to announce that Databricks was named one of the 2024 Fortune Best Workplaces in Technology™ . This award reflects our... View the full article

1GB of data was referred to as big data in 1999. Nowadays, the term is used for petabytes or even exabytes of data (1024 Petabytes), close to trillions of records from billions of people. In this fast-moving landscape, the key to making a difference is picking up the correct data storage solution for your business. […]View the full article

We’re excited to announce that Hevo Data has achieved the prestigious Snowflake Ready Technology Validation certification! This recognition solidifies our commitment to delivering top-notch data integration solutions that seamlessly work with Snowflake, a leading AI Data Cloud. What is the Snowflake Ready Technology Validation Program? The Snowflake Ready Tech Validation Program identifies and acknowledges partners […]View the full article

Data pipelines and workflows have become an inherent part of the advancements in data engineering, machine learning, and DevOps processes. With ever-increasing scales and complexity, the need to orchestrate these workflows efficiently arises. That is where Apache Airflow steps in —an open-source platform designed to programmatically author, schedule, and monitor workflows. In this blog, we […]View the full article

ETL tools have become important in efficiently handling integrated data. In this blog, we will discuss Fivetran vs AWS Glue, two influential ETL tools on the market. This will help you gain a comprehensive understanding of the product’s features, pricing models, and real-world use cases, helping you choose the right solution. Overview of Fivetran G2 […]View the full article

A Data Pipeline is an indispensable part of a data engineering workflow. It enables the extraction, transformation, and storage of data across disparate data sources and ensures that the right data is available at the right time. Python has emerged as a favorite tool for building such pipelines due to its scripting simplicity, extensive libraries, […]View the full article

In today’s competitive era, data is a catalyst fueling businesses to grow faster. As data volumes increase, fetching insights from this data comes with its challenges. Sure, you can use lakes and marts to dump any data, but ultimately, deriving business insights requires structured data with a faster querying experience. This raises the need for […]View the full article

Two platforms are most commonly associated with automating your data processes: Fivetran vs Supermetrics. Thus, whether you have the demands of a fast-paced marketing team that needs the functionality of integrating many sources into one destination or an adept technical team that has to maintain high-traffic ETL processes, deciding between Fivetran vs Supermetrics may be […]View the full article

Nowadays, when it comes to data management, every business has to make one critical decision: whether to use a Data Mesh or a Data Warehouse. Both are strong data management architectures, but they are designed to support different needs and various organizational structures. Selecting the right one can make or break how efficiently you manage […]View the full article

In Snowflake, the views are crucial for organizing, selecting, and retrieving data while not copying the data itself. Instead, if performance is a concern—such as in querying large data sets—then Snowflake materialized views are perfect. In this blog, we’ll explore: Ready? Let’s start by talking about views in general. What are Views? In simple terms, […]View the full article

We are excited to introduce several powerful new capabilities to Mosaic AI Gateway, designed to help our customers accelerate their AI initiatives with... View the full article

Imagine giving your business an intelligent bot to talk to customers. Chatbots are commonly used to talk to customers and provide them with... View the full article

Personalization and scale have historically been mutually exclusive. For all the talk of one-to-one marketing and hyper-personalization , the reality has been that... View the full article

As recently announced at this year’s Data and AI Summit, Databricks AI/BI democratizes business intelligence and analytics across your organization with highly visual... View the full article

Did you know that Netflix is one of the biggest clients for AWS? They did not just push a button when they shifted their entire data infrastructure. It took them seven years to complete the entire migration and ensure that every piece of data moved securely and perfectly into the new system. This shows us […]View the full article

Building an efficient data stack that can handle big data is no small feat, whether due to growing data demands or operational costs. A modern data stack solves these problems by automating and streamlining many data tasks, from sourcing to transformation. In this article, we will detail what a modern data stack is and its […]View the full article

Over the past three months, I had the opportunity to work as a Product Management Intern on the Ingestion team at Databricks. During... View the full article

Segmentation projects are the cornerstone of personalization in games. Personalization of the player experience helps maximize player engagement, mitigate churn and increase player... View the full article

Maintaining heavy equipment assets, such as oil rigs, agricultural combines, or fleets of vehicles, poses an extremely complex challenge for global companies. These... View the full article