Data Engineering & Data Science

Your organization might store sensitive data such as identification numbers, date of birth, or account numbers in Snowflake data warehouse tables. To ensure this information is accessible only to authorized people with appropriate roles, Snowflake supports column-level security through dynamic data masking policies. With the Snowflake data masking feature, sensitive columns within tables or views […]View the full article

The constant increase in the data produced by modern technologies has given rise to significant challenges, such as data complexity, inconsistencies, and breaching issues. You need a structured approach to address these challenges and mitigate the risk of comprising sensitive data. What is Data Governance in Snowflake? While Snowflake enables you to handle increasing volumes […]View the full article

Snowflake is a cloud-based platform that manages large data workloads in virtual warehouses. It is known for its unique architecture and pricing model. Snowflake charges you for the compute resources, storage, and data transfer services you utilize. However, most of the costs on your Snowflake bill are based on the compute resources you use rather […]View the full article

Snowflake is a cloud data warehousing solution that has become popular for companies with large data volumes. However, moving databases from an existing data platform to Snowflake can be complicated. You may face challenges in adapting existing pipelines that require custom code or integrating data from legacy systems to Snowflake’s environment. This article will provide […]View the full article

Today, we are excited to announce Databricks LakeFlow, a new solution that contains everything you need to build and operate production data pipelines... View the full article

Discover how RudderStack reduced Customer Success response times by 50% using LLMs integrated with kapa.ai and Thena View the full article

At Databricks, our mission is to democratize data + AI. An open approach to sharing and collaboration is critical to maximize reach and... View the full article

We are excited to announce that we are open sourcing Unity Catalog, the industry’s first open source catalog for data and AI governance... View the full article

In an era marked by rapid advancements in artificial intelligence and an explosion of data and Gen AI tools, enterprises face fragmented data... View the full article

The annual Data Team Awards showcase how different data teams from across the globe are delivering solutions to some of the world’s most... View the full article

“FactSet’s mission is to empower clients to make data-driven decisions and supercharge their workflows and productivity. To deliver AI-driven solutions across our entire... View the full article

Today, we are excited to announce Databricks AI/BI , a new type of business intelligence product built from the ground up to deeply... View the full article

RudderStack Profiles enables data teams to power their businesses with complete customer profiles and now supports the Databricks Data Intelligence Platform.View the full article

Generative AI fever shows no signs of cooling off. As pressure and excitement build to execute strong GenAI strategies, data leaders and practitioners... View the full article

We're excited to announce the General Availability of Databricks Predictive Optimization. This capability intelligently optimizes your table data layouts for faster queries and... View the full article

The Databricks Partner Ecosystem, comprising over 3,800 partners worldwide, plays a pivotal role in building and delivering premier data and AI solutions globally... View the full article

In the dynamic, innovative landscape of the San Francisco Bay Area, Databricks stands out not just for our groundbreaking data and AI solutions... View the full article

This is a collaborative post from Databricks and Google Cloud. We thank Nicole Huynh , Partner Marketing Manager - Data Cloud, for her... View the full article

In June 2023, we launched Databricks Marketplace as an open marketplace for all your data, analytics, and AI needs, powered by the open... View the full article

This is a collaborative post from Databricks and Microsoft. We thank Mohini Verma , Senior Product Marketing Manager, for her contributions. Data +... View the full article

This blog is authored by Bhaskar Palit , Senior Director, Data & Analytics, PepsiCo, and Sudipta Das , Data Architect Senior Manager, PepsiCo... View the full article

BigQuery, now with first-party support for Delta Lake, grows Delta Lake’s vibrant connector ecosystem and simplifies its integration with Databricks. View the full article

Introduction The Internet of Things (IoT) is generating an unprecedented amount of data. IBM estimates that annual IoT data volume will reach approximately... View the full article

We are excited to announce that Forrester has recognized Databricks as a Leader in The Forrester Wave™: AI Foundation Models for Language, Q2... View the full article

Data + AI Summit 2024 will be held in person and virtually on June 10-13, 2024, with a highly anticipated lineup of keynotes... View the full article

Introduction Today, manufacturers’ field maintenance is often more reactive than proactive, which can lead to costly downtime and repairs. Historically, data warehouses have... View the full article

The annual Data Team Awards spotlight data teams and the pivotal role they play in business operations across industries and markets. By continually... View the full article

Over the last year, we’ve been listening to feedback and iterating on new ideas with a single goal: to build the best data-focused... View the full article

We are excited to announce that we have agreed to acquire Tabular, Inc, a data management company founded by Ryan Blue, Daniel Weeks... View the full article

Businesses are making remarkable progress on their data and AI journeys. They’re advancing from a few pilot projects confined to use cases likely... View the full article

The secret to good AI is great data. As AI adoption soars, the data platform is the most important component of any enterprise's... View the full article

Delta Lake UniForm, now in GA, enables customers to benefit from Delta Lake’s industry-leading price-performance when connecting to tools in the Iceberg ecosystem. View the full article

We are excited to announce a new data type called variant for semi-structured data. Variant provides an order of magnitude performance improvements compared... View the full article

With more and more customer interactions moving into the digital domain, it's increasingly important that organizations develop insights into online customer behaviors. In... View the full article

As data continues to grow at an unprecedented rate, the need for an efficient and scalable open-source ETL solution becomes increasingly pressing. However, with every organisation’s varying needs and the cluttered market for ETL tools, finding and choosing the right tool can be strenuous. I have curated an open-source etl tools list, ranked by popularity […]View the full article

Secure data sharing is a prominent feature of Snowflake that allows you to collaborate with others in the same region. This feature enables the secure transfer of specific database objects between Snowflake accounts. As a consumer, you can access the data without worrying about local storage; you pay only for the computing resources used. Since […]View the full article

Have you ever wondered how Snowflake’s date and time functions can enhance your organization’s data analysis capabilities? Consider a scenario where you must prepare a financial report by calculating the days between the order and delivery dates for all products sold within a week. While a Snowflake minus (-) operator might suffice to subtract these […]View the full article

The exponential data growth has increased the demand for tools that make data processes, such as data collection, integration, and transformation, as smooth as possible. These tools and technologies can help you evolve your methods of handling organizational data. Among these tools, you can leverage Snowflake and dbt to address some of these crucial data […]View the full article

How often does your data team use third-party data to make business decisions? Snowflake data exchange is a good step in that direction to turn your business data-driven. It creates a data-sharing hub for organizations to become data providers or consume data from others, depending on the business use case. In this article, I will […]View the full article

“According to Statista, the total volume of data was 64.2 zettabytes in 2020; it’s predicted to reach 181 zettabytes by 2025.” In this day and age, the importance of good data collection and efficient data cleansing for better analysis has grown to become vital. The reason is straightforward: A data-driven decision is as good as […]View the full article

MySQL is a Relational Database Management System. This Open-source tool is one of the best RDBMS available in the market that is being used to develop web-based software applications among others. MySQL is scalable, intuitive, and swift when compared to its contemporaries. It houses a Client-Server architecture. At the core of the MySQL Database lies […]View the full article

With the huge volumes of data being generated by enterprises today, businesses are looking for modernized means of data storage. On-premise storage options are associated with many limitations including lack of adequate scalability, poor accessibility, and high burden on maintenance. That’s why organizations are moving their data from on-premise storage to the Cloud. This article […]View the full article

Data plays an important role in most of the decision-making processes be it related to business or even Engineering processes. It was difficult in the past to set up and maintain the analytics stack easily for it to play a pivotal role in every process hence it was seldom used, however, the advent of cloud […]View the full article

Apache Airflow is a tool that can create, organize, and monitor workflows. It is open-source hence it is free and has a wide range of support as well. It is one of the most trusted platforms that is used for orchestrating workflows and is widely used and recommended by top data engineers. This tool provides […]View the full article

Special thanks to Caleb Benningfield and Sam Malissa at Amperity for their valuable insights and contributions to this blog. Today, businesses face a... View the full article

We are excited to announce the general availability of Row Filters and Column Masks in Unity Catalog on AWS , Azure and GCP... View the full article

Salesforce and Databricks are excited to announce an expanded strategic partnership that delivers a powerful new integration - Salesforce Bring Your Own Model... View the full article

Whether you are working on a live title, pre/post production, ongoing maintenance, future releases, another version of a game, or a brand new... View the full article

This blog is authored by Michael Ewins, Director of Engineering at Skyscanner At Skyscanner , we're more than just a flight search engine... View the full article

Over the last few years, Large Language Models (LLMs) have been reshaping the field of natural language, thanks to their transformer-based architectures and... View the full article