Search the Community

Balancing correctness, latency, and cost in unbounded data processingImage created by the author.Table of contentsBefore we move onIntroduction from the paper.The details of the Dataflow model.Implementation and designs of the model.IntroGoogle Dataflow is a fully managed data processing service that provides serverless unified stream and batch data processing. It is the first choice Google would recommend when dealing with a stream processing workload. The service promises to ensure correctness and latency regardless of how big your workload is. To achieve these characteristics, Google Dataflow is backed by a dedicated processing model, Dataflow, resulting from many years of Google research and development. This blog post is my note after reading the paper: The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing. If you want to learn more about stream processing, I strongly recommend this paper. It contains all the lessons and insights from Google’s introduction of the Dataflow model to deal with its global-scale stream processing demand. Despite being written in 2015, I believe this paper’s contribution will never be old. Note: The paper was published in 2015, so some details may be changed or updated now; if you have any feedback or information that can supplement my blog, feel free to comment.Before we move onTo avoid more confusingDataflow is the Google stream processing model.Apache Beam lets users define processing logic based on the Dataflow model.Google Cloud Dataflow is a unified processing service from Google Cloud; you can think it’s the destination execution engine for the Apache Beam pipeline.Workflow: You define the unified processing logic using Apache Beam and decide to run the pipeline on the execution engine you want, such as Google Dataflow, Spark, Flink, etc. Before we explore the Dataflow model in depth, the following sections will introduce some information, such as context, challenges, and concepts.Paper’s IntroductionAt the time of the paper writing, data processing frameworks like MapReduce and its “cousins “ like Hadoop, Pig, Hive, or Spark allow the data consumer to process batch data at scale. On the stream processing side, tools like MillWheel, Spark Streaming, or Storm came to support the user. Still, these existing models did not satisfy the requirement in some common use cases. Consider an example: A streaming video provider’s business revenue comes from billing advertisers for the amount of advertising watched on their content. They want to know how much to bill each advertiser daily and aggregate statistics about the videos and ads. Moreover, they want to run offline experiments over large amounts of historical data. They want to know how often and for how long their videos are being watched, with which content/ads, and by which demographic groups. All the information must be available quickly to adjust their business in near real-time. The processing system must also be simple and flexible to adapt to the business’s complexity. They also require a system that can handle global-scale data since the Internet allows companies to reach more customers than ever. Here are some observations from people at Google about the state of the data processing systems of that time: Batch systems such as MapReduce, FlumeJava (internal Google technology), and Spark fail to ensure the latency SLA since they require waiting for all data input to fit into a batch before processing it.Streaming processing systems that provide scalability and fault tolerance fall short of the expressiveness or correctness aspect.Many cannot provide exactly once semantics, impacting correctness.Others lack the primitives necessary for windowing or provide windowing semantics that are limited to tuple- or processing-time-based windows (e.g., Spark Streaming)Most that provide event-time-based windowing rely on ordering or have limited window triggering.MillWheel and Spark Streaming are sufficiently scalable, fault-tolerant, and low-latency but lack high-level programming models.They conclude the major weakness of all the models and systems mentioned above is the assumption that the unbounded input data will eventually be complete. This approach does not make sense anymore when faced with the realities of today’s enormous, highly disordered data. They also believe that any approach to solving diverse real-time workloads must provide simple but powerful interfaces for balancing the correctness, latency, and cost based on specific use cases. From that perspective, the paper has the following conceptual contribution to the unified stream processing model: Allowing for calculating event-time ordered (when the event happened) results over an unbounded, unordered data source with configurable combinations of correctness, latency, and cost attributes.Separating pipeline implementation across four related dimensions:- What results are being computed? - Where in event time they are being computed. - When they are materialized during processing time, - How do earlier results relate to later refinements?Separating the logical abstraction of data processing from the underlying physical implementation layer allows users to choose the processing engine.In the rest of this blog, we will see how Google enables this contribution. One last thing before we move to the next section: Google noted that there is “nothing magical about this model. “ The model doesn’t make your expensive-computed task suddenly run faster; it provides a general framework that allows for the simple expression of parallel computation, which is not tied to any specific execution engine like Spark or Flink. Unbounded/BoundedImage created by the author.The paper’s authors use the term unbounded/bounded to define infinite/finite data. They avoid using streaming/batch terms because they usually imply using a specific execution engine. The term unbound data describes the data that doesn’t have a predefined boundary, e.g., the user interaction events of an active e-commerce application; the data stream only stops when the application is inactive. Whereas bounded data refers to data that can be defined by clear start and end boundaries, e.g., daily data export from the operation database. To continue with the introduction section, we will review some concepts used throughout the paper.WindowingThe organizerWindowing divides the data into finite chunks. Usually, the system uses time notions to organize data into the window (e.g., all data in the last 1 hour will belong to one window). All data in the windows are processed as a group. Users require grouping operations on the window abstractions: aggregation or time-bounded operation when processing unbound data. On the other hand, some operations on unbounded data don’t need the window notion, like filtering, mapping, or inner join. Windows may be aligned, e.g., applied across all the data for a given window, or unaligned, e.g., applied across only specific subsets of the data in that window. There are three major types of windows: Image created by the author.Fixed: The windows are defined as static window size, e.g., hourly windows.Sliding: The windows are defined by a window size and slide period, e.g., 30-minute windows starting every five minutes.Sessions: The windows capture some period of activity over a subset of the data, in this case, per key. Typically, they are defined by a timeout gap.Time DomainsWhen handling time-related events data, there are two domains of time to consider: Event Time: the time the event itself happened. For example, if the system device recorded you purchasing a game item at 11:30, this is considered the event time.Processing Time: The time at which an event is observed at any given point during processing. For example, the purchased game item is recorded at 11:30 but only arrives at the stream processing system at 11:35; this “11:35“ is the processing time.Given that definition, event time will never change, but processing time changes constantly for each event as it flows through the pipeline step. This is a critical factor when analyzing events in the context of when they occurred. The difference between the event_time and the processing_time is called time domain skew. The skew can result from many potential reasons, such as communication delays or time spent processing in each pipeline stage. Metrics, such as watermarks, are good ways to visualize the skew. For the paper, the authors considered a lower watermark on event times that the pipeline has processed. These watermarks provide a notion to tell the system that: “no more data which have event time sooner this point of time will appear in the pipeline.” the watermarks are used not only to observe the skew between time domains but also to monitor the overall system. In a super-ideal world, the skew would always be zero; we could always process all events right when they happen. Image created by the author.In the following sections, we will learn the details of the Dataflow model.Core primitivesThe model has two core transformations that operate on the (key, value) pair; both transformations can work on bounded and unbounded data: Image created by the author.ParDo is for generic parallel processing. It will process each input element with a provided user-defined function (called a DoFn in Dataflow), which can produce zero or more output per input element. The input doesn’t need to be the unbound collections.GroupByKey for grouping operations based on the defined key.The ParDo operates on each element so it can be translated to unbounded data. The GroupByKey collects all data for a given key before sending it to the downstream steps. If the input source is unbounded, it is impossible to define when it will end. The standard solution is data windowing. WindowingSystems that support grouping typically redefine their GroupByKey operation to be GroupByKeyAndWindow. The authors' significant contribution in this aspect is the unaligned window. The first is treating all windowing strategies as unaligned from the dataflow model and allowing custom adjustments to apply aligned windows when needed. The second is any windowing process can be broken apart into two related operations: Image created by the author.AssignWindows assigns the element to zero or more windows. From the model’s view, window assignment creates a new copy of a component in each window.MergeWindows merges windows at grouping time. This allows the construction of data-driven windows over time as data arrive and are grouped. Window merging occurs as part of the GroupByKeyAndWindow operation. We see the example below for a better understanding:Triggers & Incremental ProcessingAlthough there is support for unaligned windows, event-time windows raised another challenge: The need to tell the system when to emit the results for a window because the data can appear in the pipeline in an unordered way. The initial solution of using event-time progress metrics like watermark (which is mentioned above) has some shortcomings: A reminder so you don’t have to scroll up: The watermark is an indicator that tells the system that “no more data which have event time sooner this point of time will appear in the pipeline.” For example, at the given time, the watermark is “11:30”, meaning no events with event_time less than 11:30 will appear anymore.They are sometimes too fast: this behavior means late data may arrive behind the watermark.They are sometimes too slow: this behavior can cause the whole pipeline to be held back to wait for a slow data point.This led to the observation that using only watermarks to decide when to emit the window’s result is likely to increase the latency (when the watermark is slow) or impact the accuracy of the pipeline (missing some data if the watermark is too fast ). The authors observe in the Lambda Architecture (which has two separate pipelines, streaming and batch, and the result from the two pipelines finally converge in the end) that the paradigm doesn’t solve the completeness problem by providing correct answers faster; instead, it gives the low-latency estimate of a result from the streaming pipeline, then promises to deliver the correctness result from the batch pipeline. They stated that if we want to achieve the same thing in a single pipeline, we need a mechanism to provide multiple panes (answers) for any given window. This feature, called trigger, allows the user to specify when to trigger the output results for a given window. Here is an illustration to provide you with a similar idea between the trigger and the semantics in Lambda Architecture Image created by the author.The system the authors introduce supports the following trigger implementation: Image created by the author.Triggering at completion estimates such as watermarks.Triggering at the point in processing time.Triggering based on data-arriving characteristics such as counts, bytes, data punctuations, pattern matching, etc.Supporting the implementation combination using loops, sequences, or logical combinations (AND, OR)The users can define their triggers utilizing both the underlying primitives of the execution runtime (e.g., watermark timers, processing-time timers) and external signals (e.g., data injection requests, external progress metrics)Besides controlling when the system will emit the window’s result, the trigger mechanism also provides a way to control how panes (answers) for a given window relate to each other via the following refinement modes: Image created by the author.Discarding: When triggering, the system discards all content’s window. The later results have no relation to previous results. This mode is helpful in cases where the downstream consumer needs the values from various triggers to be independent. This is also the most efficient option in terms of space for buffering data.Accumulating: When triggering, the system keeps window contents in a persistent state; later results are related to previous results. This is useful when the downstream consumer expects to overwrite old values with new ones when receiving multiple results for the same window. It is also the mode used in Lambda Architecture systems, where the streaming pipeline outputs low-latency results, which are then overwritten later by the results from the batch pipeline.Accumulating & Retracting: When triggering, in addition to the Accumulating semantics, the emitted result’s copy is also stored in a persistent state. When the window triggers again in the future, a retraction for the previous value will be emitted first, followed by the new value.The following section will describe how Google implements and designs the Dataflow model.ImplementationThe paper’s authors say they’ve implemented this model internally using FlumeJava, a Java library that makes it easy to develop, test, and run efficient data-parallel pipelines. MillWheel acts as the beneath stream execution engine. Additionally, an external reimplementation for Google Cloud Dataflow is primarily complete at the time of the paper’s writing. Interestingly, the core windowing and triggering code is quite general, and a significant portion is shared across batch and streaming implementations. Design PrinciplesThe core principles of the Dataflow model: Never rely on any notion of completeness.Be flexible to accommodate the diversity of known use cases and those to come in the future.It not only makes sense but also adds value in the context of each of the envisioned execution engines.Encourage clarity of implementation.Support robust analysis of data in the context in which they occurred.Motivating ExperiencesAs they designed the Model, they gained real-world experiences with FlumeJava and MillWheel. Things that worked well would be reflected in the model; things that were less well would motivate changes in approach. Here are some of their experiences that influenced the design choice: Unified Model: The original motivation for this design choice is that one huge pipeline runs in streaming mode on MillWheel by default but with a dedicated FlumeJava batch implementation for large-scale backfills. Another motivation came from an experience with Lambda Architecture, where one customer ran the streaming pipeline in MillWheel with a nightly MapReduce (batch) to generate truth. They found that customers stopped trusting the weakly consistent results between pipelines over time.Sessions are a critical use case within Google. This mechanism is used in many cases, including search, ads, analytics, social media, and YouTube. Any users who care about correlating bursts of user activity over a period of time would leverage sessions. Thus, support for sessions became an indispensable part of the model’s design.Triggers, Accumulation, & Retraction: Two teams with billing pipelines running on MillWheel had problems that motivated parts of the model. The best practice at the time was to use the watermark as a completion metric, with extra ad hoc logic for late data. Lacking a system for updates and retractions, a team that processed resource utilization statistics decided to build their own solution. Another billing team had significant issues with watermark lags caused by stragglers (slow-running units affect overall job performance completion.) in the input. These shortcomings became significant motivators in the design and shifted the focus from targeting completeness to adaptability over time. This results in two decisions: triggers, which allow the flexible specification of when results are materialized, and incremental processing support via accumulation.Watermark Triggers: Many MillWheel pipelines calculate aggregate statistics. Most do not require 100% accuracy; they care about having a mostly complete view of their data in a reasonable amount of time. Given the high level of accuracy that they achieve with watermarks for structured input sources like log files, customers find watermarks very effective in triggering a single, highly accurate aggregate per window.Processing Time Triggers: The recommendation pipelines emit their output using processing-time timers. These systems, having regularly updated, partial views of the data, were much more valuable than waiting until mostly complete views were ready based on the watermark. This also meant that the notion of a watermark would not affect the timeliness of output for the rest of the data.Data-Driven & Composite Triggers: The different detection systems in the anomaly detection pipeline used to track trends in Google web search motivated the data-driven triggers. These differences observe the stream of queries and calculate statistical estimates to check whether a spike exists. When they believe a spike is happening, they emit a start record; when they think it has ceased, they emit a stop. It was also a motivating case for trigger composition because, in reality, the system runs multiple differs simultaneously, multiplexing the output according to a set of logic.OutroIn this week’s blog, we’ve discussed the design principle and implementation of the Dataflow model, the backbone behind the famous Google Cloud Dataflow service. If you want to dive deeper into the model, I highly recommend the book Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing or the two-part blog from one of the paper’s authors: Streaming 101 and Streaming 102. I hope my work brings some value, especially to someone who wants to learn more about the stream processing world. See you next blog! References[1] Google, The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data (2015). My newsletter is a weekly blog-style email in which I note things I learn from people smarter than me. So, if you want to learn and grow with me, subscribe here: https://vutr.substack.com. The Stream Processing Model Behind Google Cloud Dataflow was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story. View the full article

In today’s data-driven world, data storage and analysis are essential to derive deeper insights for smarter decision-making. As data volumes increase, organizations consider shifting transactional data from Oracle databases on AWS RDS to a powerful platform like Google BigQuery. It can be due to several reasons, which include AWS RDS Oracle’s storage limits, high query […]View the full article

Most businesses know that taking responsibility for their environmental and social impact is key for long-term success. But how can they make fully-informed decisions when most companies only have visibility into their immediate suppliers? At Prewave, we’re driven by the mission to help companies make their entire supply chains more resilient, transparent, and sustainable. Our end-to-end platform monitors and predicts a wide range of supply chain risks, and AI is the driving-force behind its success. Without AI, handling vast volumes of data and extracting meaningful insights from publicly-available information would be almost unfathomable at the scale that we do to help our clients. Because of that, Prewave needs a rock-solid technology foundation that is reliable, secure, and highly scalable to continually handle this demand. That’s why we built the Prewave supply chain risk intelligence platform on Google Cloud from inception in 2019. Back then, as a small team, we didn’t want to have to maintain hardware or infrastructure, and Google Cloud managed services stood out for providing reliability, availability, and security while freeing us up to develop our product and focus on Prewave’s mission. A shared concern for sustainability also influenced our decision, and we’re proud to be working with data centers with such a low carbon footprint. Tracking hundreds of thousands of suppliers Prewave’s end-to-end platform solves two key challenges for customers: First, it makes supply chains more resilient by identifying description risks and developing the appropriate mitigation plans. And second, it makes supply chains more sustainable by detecting and solving ESG risks, such as forced labor or environmental issues. It all starts with our Risk Monitoring capability, which uses AI that was developed by our co-founder Lisa in 2012 during her PhD research. With it, we’re scanning publicly available information in 120+ languages, looking for insights that can indicate early signals of Risk Events for our clients, such as labor unrest, an accident, fire, or 140 other different risk types that can disrupt their supply chain. Based on the resulting insights, clients can take actions on our platform to mitigate the risk, from filing an incident review to arranging an on-site audit. With this information, Prewave also maps our clients’ supply chains from immediate and sub-tier suppliers down to the raw materials’ providers. Having this level of granularity and transparency is now a requirement of new regulations such as the European corporate sustainability due diligence directive (CSDDD), but it can be challenging for our clients to do without help. They usually have hundreds or thousands of suppliers and our platform helps them to know each one, but also to focus attention, when needed, on those with the highest risk. The Prewave platform keeps effort on the supplier’s side as light as possible. They only have to act if potential risk is flagged by our Tier-N Monitoring capability, in which case, we support them to fix issues and raise their standards. Additionally, this level of visibility frees them up from having to manually answer hundreds of questionnaires in order to qualify to do business with more partners. To make all this possible, our engineering teams rely heavily on scalable technology such as Google Kubernetes Engine (GKE) to support our SaaS. We recently switched from Standard to Autopilot and noticed great results in time efficiency now that we don’t need to ensure that node pools are in place or that all CPU power available is being used appropriately, helping save up to 30% of resources. This also has helped us to reduce costs because we only pay for the deployments we run. We also believe that having the best tools in place is key to delivering the best experience not only to customers but also to our internal teams. So we use Cloud Build and Artifact Registry to experiment, build, and deploy artifacts and manage docker containers that we also use for GKE. Meanwhile, Cloud Armor acts as a firewall protecting us against denial of service and web attacks. Because scalability is key for our purposes, the application development and data science teams use Cloud SQL as a database. This is a fully managed service that helps us focus on developing our product, since we don’t have to worry about managing the servers according to demand. Data science teams also use Compute Engine to host our AI implementations as we develop and maintain our own models, and these systems are at the core of Prewave’s daily work. Helping more businesses improve their deep supply chains Since 2020, Prewave has grown from three clients to more than 170, our team of 10 grew to more than 160, and the company’s revenue growth multiplied by 100, achieving a significant milestone. We’ve also since then released many new features to our platform that required us to scale the product alongside scaling the company. With Google Cloud, this wasn’t an issue. We simply extended the resources that the new implementations needed, helping us to gain more visibility at the right time and win new customers. Because our foundation is highly stable and scalable, growing our business has been a smooth ride. Next, Prewave is continuing its expansion plans into Europe that began in 2023, before moving to new markets, such as the US. This is going well and our association with Google Cloud is helping us win the trust of early-stage clients who clearly also trust in its reliability and security. We’re confident that our collaboration with Google Cloud will continue to bring us huge benefits as we help more companies internationally to achieve transparency, resilience, sustainability, and legal compliance along their deep supply chains. View the full article

Google Cloud Champion Innovators are a global network of more than 600 non-Google professionals, who are technical experts in Google Cloud products and services. Each Champion specializes in one of nine different technical categories which are cloud AI/ML, data analytics, hybrid multi-cloud, modern architecture, security and networking, serverless app development, storage, Workspace and databases. In this interview series we sit down with Champion Innovators across the world to learn more about their journeys, their technology focus, and what excites them. Today we’re talking to Juan Guillermo Gómez. Currently Technical Lead at Wordbox, Juan Guillermo is a Cloud Architect, Google Developer Expert, Serverless App Development Champion, and community builder who is regularly invited to share his thoughts on software architecture, entrepreneurship, and innovation at events across Latin America. Natalie Tack (Google Cloud Editorial): What technology area are you most fascinated with, and why? Juan Guillermo Gómez: As a kid, I dreamed of developing software that would change the world. I've always been fascinated by programming and the idea of creating products and services that make people's lives easier. Nowadays, my focus is on architecture modernization and using innovative new tech to scale and improve systems. Following a Google Cloud hackathon around 10 years ago, I started to take an interest in serverless architecture and became really enthusiastic about Google Cloud serverless computing services, which enable you to focus on coding without worrying about infrastructure. Nowadays, I’m excited about the potential of AI to help us create better, more robust, and more efficient systems, and I'm really looking forward to seeing where that will take us. NT: As a developer, what’s the best way to go about learning new things? JGG: There’s a wealth of resources out there, whether it’s YouTube, podcasts or developer blogs. I find the Google Cloud developer blog and YouTube channel particularly instructive when it comes to real use cases. But the most important thing in my opinion is to be part of a community. It enables you to share experiences, collaborate on projects and learn from others with expertise in specific industries. Google Developer Groups, or GDGs, for example are a great way to network and keep up with the latest developments. Becoming a Google Cloud Champion Innovator has been really beneficial. It enables me to learn directly from Googlers, collaborate around shared problems, and work directly with other people from my field. I can then share that knowledge with my community in Latin America, both as co-organizer of GDG Cali and via my podcast Snippets Tech. NT: Could you tell us more about your experience as an Innovator? JGG: I joined the Innovators program in September 2022, and it was a natural fit from the start. The Innovator culture is deeply rooted in that of developer communities, which I’ve participated in for over 20 years now. The core philosophy is sharing, collaboration and learning from others' experiences, not only through getting together at talks and conferences, but also open source software and libraries. Basically, creating things that benefit the community at large: the Innovator mindset is fundamentally collaborative. As a result, the program provides a wealth of information, and one of the biggest benefits from my point of view is the amount of real use cases it gives access to. I’ve learnt a lot about embedding and semantic similarity that I’m putting into practice in my work as Technical Lead at Wordbox, a startup that helps people learn languages through TV and music. NT: What are some upcoming trends you’re enthusiastic about? JGG: Generally speaking, I’m very interested to see where we’ll go with generative AI. I started working with Google Cloud APIs such as Translate and Speech-to-Text around three years ago as part of my work with Wordbox, and I’m impressed with the way Google Cloud democratizes machine learning and AI, allowing anyone to work with it without extensive machine learning knowledge. There are so many potential use cases for gen AI. As a developer, you can work faster, solve programming language challenges, write unit tests, and refer to best practices. As an Innovator, I gained early access to Duet AI for Developers (now called Gemini Code Assist), which is a great tool to fill in knowledge gaps, suggest scripts and help out junior architects or those that are new to Google Cloud - basically helping you focus on creating great code. NT: Can you tell us about an exciting new project you’re working on? JGG: The Wordbox language learning app features a collection of short videos with English phrases that we show users based on semantic similarity, where the phrase in the new video is similar to the previous one. To enable that, we use Vertex AI, PaLM 2 and Vector Search, and are keen to explore Gemini models, as they offer advanced capabilities for comparing semantic similarities not only between text, but also between text and video, which would enable us to create stories around specific phrases. For example, if a user is watching a video related to a “Game of Thrones” review series and learning certain expressions from it, we can use Gemini models to find similarities with other videos or texts. This will allow us to produce a narrative around the learned expression, creating a comprehensive learning environment that’s tailored to the user’s proficiency level. The learner can then read or watch the story, answer questions and engage in a more interactive, personalized learning experience. As a side project, I’m also working on an AI-enabled platform that helps musicians and singers create lyrics based on keywords, genre, and context. They input the information, and the platform generates lyrics that hopefully serve as a jumping-off point for a great new piece of music. NT: What advice would you give to budding innovators? JGG: The Innovators program can be summed up in three words: networking, learning, and growth. I’d advise anyone interested in becoming an Innovator to be proactive both in learning and in engaging with their fellow developers. It feels pretty great to be given early access to an API and then a few months later tell your community about it while creating fantastic new features for your clients. Take the next steps on your Google Cloud journey and learn more about the Google Cloud Innovators Program, designed to help developers and practitioners grow their Google Cloud skills and advance their careers. No matter where you are on your cloud journey, the Innovators Program has something for you! View the full article

Today, we're launching the general availability (GA) of Direct VPC egress for Cloud Run. This feature enables your Cloud Run resources to send traffic directly to a VPC network without proxying it through Serverless VPC Access connectors, making it easier to set up, faster, and with lower costs. In fact, Direct VPC egress delivers approximately twice the throughput compared to both VPC connectors and the default Cloud Run internet egress path, offering up to 1 GB per second per instance. Whether you're sending traffic to destinations on the VPC, to other Google Cloud services like Cloud Storage, or to other destinations on the public internet, Direct VPC egress offers higher throughput and lower latency for performance-sensitive apps. What's new since the preview Notable improvements and new features: All regions where Cloud Run is available are now enabled for Direct VPC egress. Each Cloud Run service revision with Direct VPC can now scale beyond 100 instances as controlled by a quota. There is a standard quota increase request process if you need to scale even more. Cloud NAT is supported, and Direct VPC egress traffic is now included in VPC Flow Logs and Firewall Rules Logging. These updates address the top issues reported by our preview customers, especially larger customers with advanced scalability, networking, and security requirements. Customer feedback Many customers have been trying Direct VPC egress in preview since last year and have given us great feedback, including DZ BANK: "With Direct VPC egress for Cloud Run, the platform team can more easily onboard new Cloud Run workloads because we no longer need to maintain Serverless VPC Access connectors and their associated dedicated /28 subnets. In our dynamic environment, where new Cloud Run services are created regularly, this simpler networking architecture saves us 4-6 hours per week of manual toil. We have also deprovisioned 30+ VPC connectors, saving on the additional compute costs for running them." - Tim Harpe, Senior Cloud Engineer, DZ BANK If you enable direct VPC egress and send all your egress traffic to a VPC, you can leverage the same tools and capabilities for all your traffic – from Cloud Run, GKE, or VMs. Next steps Direct VPC egress is ready for your production workloads. Try it today and enjoy better performance and lower cost. For a primer about how Direct VPC egress works, check out our preview blog post and its attached explainer video. View the full article

At Ninja Van, running applications in a stable, scalable environment is business-critical. We are a fast-growing tech-enabled express logistics business with operations across South East Asia. In the past 12 months, we’ve partnered with close to two million businesses to deliver around two million parcels daily to customers, with the help of more than 40,000 workers and employees. We are an established customer of Google Cloud, and continue to work closely with them to expand into new products and markets, including supply chain management and last mile courier services. Deploying a secure, stable, container platform To run the applications and microservices architecture that enable our core businesses, we opted to deploy a secure and stable container platform. We had been early adopters of containerization. We were initially running our container workload in CoreOS with a custom scheduler, Fleet. As we monitor the activities in the open source community, it was evident Kubernetes was gaining more traction and is becoming more and more popular. We decided to run our own Kubernetes cluster and API server, later deploying a number of Compute Engine virtual machines, consisting of both control plane and worker nodes. As we dived deeper into Kubernetes, we realized that a lot of what we did was already baked into its core functionalities, such as service discovery. This feature enables applications and microservices to communicate with each other without the need to know where the container is deployed to among the worker nodes. We felt that if we continued maintaining our discovery system, we would just be reinventing the wheel. Thus, we dropped what we had in favor of this Kubernetes core feature. We also found the opportunity to engage with and contribute to the open source community working on Kubernetes compelling. With Kubernetes being open standards-based, we gained the freedom and flexibility to adapt our technology stack to our needs. However, we found upgrading a self-managed Kubernetes challenging and troublesome in the early days and decided to move to a fully managed Kubernetes service. Among other benefits, we could upgrade Kubernetes easily through Google Kubernetes Engine (GKE) user interface or Google Cloud command line tool. We could also enable auto upgrades simply by specifying an appropriate release channel. Simplifying the operation of multiple GKE clusters With a technology stack based on open-source technologies, we have simplified the operation of multiple clusters in GKE. For monitoring and logging operations, we have moved to a centralized architecture to colocate the technology stack on a single management GKE cluster. We use Elasticsearch, Fluentbit and Kibana for logging and Thanos, Prometheus and Grafana for monitoring. When we first started, logging and monitoring were distributed across individual clusters, which meant that administrators had to access the clusters separately which led to a lot of operational inefficiencies. This also meant maintaining duplicated charts across different instances of Kibana and Grafana. For CI/CD, we run a dedicated DevOps GKE cluster used for hosting developer toolsets and running the pipelines. We embrace the Atlassian suite of services, such as JIRA, Confluence, Bitbucket, Bamboo to name a few. These applications are hosted in the same DevOps GKE cluster. Our CI/CD is centralized, but custom steps can be put in, giving some autonomy to teams. When a team pushes out new versions of codes, our CI pipeline undertakes the test and build processes. This then produces container image artifacts for backend services and minified artifacts for frontend websites. Typically, the pipelines are parameterized to push out and test the codes in development and testing environments, and subsequently deploy them into sandbox and production. Depending on their requirements, application teams have the flexibility to opt for a fully automated pipeline or a semi-automated one, that requires manual approval for deployment to production. Autoscaling also comes into play during runtime. As more requests come in, we need to scale up to more containers, but shrink down automatically as the number of requests decreases. To support autoscaling based on our metrics, we integrate KEDA to our technology stack. Delivering a seamless development experience In a sense, our developers are our customers as well as our colleagues. We aim to provide a frictionless experience for them by automating the testing, deployment, management and operation of application services and infrastructure. By doing this, we allow them to focus on building the application they’re assigned. To do this, we’re using DevOps pipelines to automate and simplify infrastructure provisioning and software testing and release cycles. Teams can also self-serve and spin up GKE clusters with the latest environment builds mirroring production with non-production sample data preloaded, which gives them a realistic environment to test the latest fixes and releases. As a build proceeds to deployment, they can visit a Bamboo CI/CD Console to track progress. Code quality is critical to our development process. An engineering excellence sub-vertical within our business monitors code quality through part of our CI/CD using the SonarQube open-source code inspection tool. We set stringent unit test coverage percentage requirements and do not allow anything into our environment that fails unit or integration testing. We release almost once a day excluding code freezes on days like Black Friday or Cyber Monday, when we only release bug fixes or features that need to be deployed urgently during demand peaks. While we’ve not changed our release schedule with GKE, we’ve been able to undertake concurrent builds in parallel environments, enabling us to effectively manage releases across our 200-microservice architecture. Latency key to Google Cloud and GKE selection When we moved from other cloud providers to Google Cloud and GKE, the extremely low latency between Google Cloud data centers, in combination with reliability, scalability and competitive pricing, gave us confidence Google Cloud was the right choice. In a scenario with a lot of people using the website, we can provide a fast response time and a better customer experience. In addition, Google Cloud makes the patching and upgrading of multiple GKE clusters a breeze by automating the upgrade process and proactively informing us when an automated upgrade or change threatens to break one of our APIs. Google Cloud and GKE also open up a range of technical opportunities for our business, including simplification. Many of our services use persistent storage, and GKE provides a simple way to automatically deploy and manage them through their Container Storage Interface (CSI) driver. The CSI driver enables native volume snapshots and in conjunction with Kubernetes CronJob, where we can easily take automated backups of the disks running services such as TiDB, MySQL, Elasticsearch and Kafka. On the development front, we are also exploring Skaffold, which opens up possibilities for development teams to improve their inner development loop cycle and develop more effectively as if their local running instance is deployed within a Kubernetes cluster. Overall, the ease of management of GKE means we can manage our 200-microservice architecture in 15 clusters with just 10 engineers even as the business continues to grow. If you want to try this out yourself, check out a hands-on tutorial where you will set up a continuous delivery pipeline in GKE that automatically rebuilds, retests, and redeploys changes to a sample application. View the full article

We've just added the new service status forums for Amazon Web Services (AWS) & Google Cloud Platform (GCP) https://devopsforum.uk/forum/32-aws-service-status/ https://devopsforum.uk/forum/33-gcp-service-status/ We've also added a new 'block' on the right-hand side of the main homepage, showing the latest 3 statuses 'Cloud Service Status' We've also added an Azure service status forum, however so far there are no posts - apparently If everything is running smoothly on Azure the feed will be empty https://azure.status.microsoft/en-gb/status/feed/ Here are some other status feeds for Azure services; https://azure.status.microsoft/en-us/status/history/ https://feeder.co/discover/d3ca207d93/azure-microsoft-com-en-us-status

Google has always pioneered the development of large and scalable infrastructure to support its search engine and other products. Its vast network servers have enabled us to store and manage immense volumes of data. As cloud computing gained notoriety, Google expanded its operations and launched Google Cloud Platform (GCP). The Google Cloud Storage (GCS) allows […]View the full article

Azure Container Apps Azure container apps is a fully managed Kubernetes service that could be compared to ECS in AWS or Cloud Run in GCP. Compared to AKS, all integrations with Azure are already done for you. The best example is the use of managed identity where here you only need to enable a parameter whereas in AKS it’s complicated and changes every two years. View the full article

Embeddings represent real-world objects, like entities, text, images, or videos as an array of numbers (a.k.a vectors) that machine learning models can easily process. Embeddings are the building blocks of many ML applications such as semantic search, recommendations, clustering, outlier detection, named entity extraction, and more. Last year, we introduced support for text embeddings in BigQuery, allowing machine learning models to understand real-world data domains more effectively and earlier this year we introduced vector search, which lets you index and work with billions of embeddings and build generative AI applications on BigQuery. At Next ’24, we announced further enhancement of embedding generation capabilities in BigQuery with support for: Multimodal embeddings generation in BigQuery via Vertex AI’s multimodalembedding model, which lets you embed text and image data in the same semantic space Embedding generation for structured data using PCA, Autoencoder or Matrix Factorization models that you train on your data in BigQuery Multimodal embeddings Multimodal embedding generates embedding vectors for text and image data in the same semantic space (vectors of items similar in meaning are closer together) and the generated embeddings have the same dimensionality (text and image embeddings are the same size). This enables a rich array of use cases such as embedding and indexing your images and then searching for them via text. You can start using multimodal embedding in BigQuery using the following simple flow. If you like, you can take a look at our overview video which walks through a similar example. Step 0: Create an object table which points to your unstructured data You can work with unstructured data in BigQuery via object tables. For example, if you have your images stored in a Google Cloud Storage bucket on which you want to generate embeddings, you can create a BigQuery object table that points to this data without needing to move it. To follow along the steps in this blog you will need to reuse an existing BigQuery CONNECTION or create a new one following instruction here. Ensure that the principal of the connection used has the ‘Vertex AI User’ role and that the Vertex AI API is enabled for your project. Once the connection is created you can create an object table as follows: code_block <ListValue: [StructValue([('code', "CREATE OR REPLACE EXTERNAL TABLE\r\n `bqml_tutorial.met_images`\r\nWITH CONNECTION `Location.ConnectionID`\r\nOPTIONS\r\n( object_metadata = 'SIMPLE',\r\n uris = ['gs://gcs-public-data--met/*']\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e9805ba60a0>)])]> In this example, we are creating an object table that contains public domain art images from The Metropolitan Museum of Art (a.k.a. “The Met”) using a public Cloud Storage bucket that contains this data. The resulting object table has the following schema: Let’s look at a sample of these images. You can do this using a BigQuery Studio Colab notebook by following instructions in this tutorial. As you can see, the images represent a wide range of objects and art pieces. Image source: The Metropolitan Museum of Art Now that we have the object table with images, let’s create embeddings for them. Step 1: Create model To generate embeddings, first create a BigQuery model that uses the Vertex AI hosted ‘multimodalembedding@001’ endpoint. code_block <ListValue: [StructValue([('code', "CREATE OR REPLACE MODEL\r\n bqml_tutorial.multimodal_embedding_model REMOTE\r\nWITH CONNECTION `LOCATION.CONNNECTION_ID`\r\nOPTIONS (endpoint = 'multimodalembedding@001')"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e9805ba6970>)])]> Note that while the multimodalembedding model supports embedding generation for text, it is specifically designed for cross-modal semantic search scenarios, for example, searching images given text. For text-only use cases, we recommend using the textembedding-gecko@ model instead. Step 2: Generate embeddings You can generate multimodal embeddings in BigQuery via the ML.GENERATE_EMBEDDING function. This function also works for generating text embeddings (via textembedding-gecko model) and structured data embeddings (via PCA, AutoEncoder and Matrix Factorization models). To generate embeddings, simply pass in the embedding model and the object table you created in previous steps to the ML.GENERATE_EMBEDDING function. code_block <ListValue: [StructValue([('code', "CREATE OR REPLACE TABLE `bqml_tutorial.met_image_embeddings`\r\nAS\r\nSELECT * FROM ML.GENERATE_EMBEDDING(\r\n MODEL `bqml_tutorial.multimodal_embedding_model`,\r\n TABLE `bqml_tutorial.met_images`)\r\nWHERE content_type = 'image/jpeg'\r\nLimit 10000"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e9805ba6610>)])]> To reduce the tutorial’s runtime, we limit embedding generation to 10,000 images. This query will take 30 minutes to 2 hours to run. Once this step is completed you can see a preview of the output in BigQuery Studio. The generated embeddings have a dimension of 1408. Step 3 (optional): Create a vector index on generated embeddings While the embeddings generated in the previous step can be persisted and used directly in downstream models and applications, we recommend creating a vector index for improving embedding search performance and enabling the nearest-neighbor query pattern. You can learn more about vector search in BigQuery here. code_block <ListValue: [StructValue([('code', "-- Create a vector index on the embeddings\r\n\r\nCREATE OR REPLACE VECTOR INDEX `met_images_index`\r\nON bqml_tutorial.met_image_embeddings(ml_generate_embedding_result)\r\nOPTIONS(index_type = 'IVF',\r\n distance_type = 'COSINE')"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e9805ba62e0>)])]> Step 4: Use embeddings for text-to-image (cross-modality) search You can now use these embeddings in your applications. For example, to search for “pictures of white or cream colored dress from victorian era” you first embed the search string like so: code_block <ListValue: [StructValue([('code', '-- embed search string\r\n\r\nCREATE OR REPLACE TABLE `bqml_tutorial.search_embedding`\r\nAS\r\nSELECT * FROM ML.GENERATE_EMBEDDING(\r\n MODEL `bqml_tutorial.multimodal_embedding_model`,\r\n (\r\n SELECT "pictures of white or cream colored dress from victorian era" AS content\r\n )\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e9805ba6850>)])]> You can now use the embedded search string to find similar (nearest) image embeddings as follows: code_block <ListValue: [StructValue([('code', '-- use the embedded search string to search for images\r\n\r\nCREATE OR REPLACE TABLE\r\n `bqml_tutorial.vector_search_results` AS\r\nSELECT\r\n base.uri AS gcs_uri,\r\n distance\r\nFROM\r\n VECTOR_SEARCH( TABLE `bqml_tutorial.met_image_embeddings`,\r\n "ml_generate_embedding_result",\r\n TABLE `bqml_tutorial.search_embedding`,\r\n "ml_generate_embedding_result",\r\n top_k => 5)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e980766d250>)])]> Step 5: Visualize results Now let’s visualize the results along with the computed distance and see how we performed on the search query “pictures of white or cream colored dress from victorian era”. Refer the accompanying tutorial on how to render this output using a BigQuery notebook. Image source: The Metropolitan Museum of Art The results look quite good! Wrapping up In this blog, we demonstrated a common vector search usage pattern but there are many other use cases for embeddings. For example, with multimodal embeddings you can perform zero-shot classification of images by converting a table of images and a separate table containing sentence-like labels to embeddings. You can then classify images by computing distance between images and each descriptive label’s embedding. You can also use these embeddings as input for training other ML models, such as clustering models in BigQuery to help you discover hidden groupings in your data. Embeddings are also useful wherever you have free text input as a feature, for example, embeddings of user reviews or call transcripts can be used in a churn prediction model, embeddings of images of a house can be used as input features in a price prediction model etc. You can even use embeddings instead of categorical text data when such categories have semantic meaning, for example, product categories in a deep-learning recommendation model. In addition to multimodal and text embeddings, BigQuery also supports generating embeddings on structured data using PCA, AUTOENCODER and Matrix Factorization models that have been trained on your data in BigQuery. These embeddings have a wide range of use cases. For example, embeddings from PCA and AUTOENCODER models can be used for anomaly detection (embeddings further away from other embeddings are deemed anomalies) and as input features to other models, for example, a sentiment classification model trained on embeddings from an autoencoder. Matrix Factorization models are classically used for recommendation problems, and you can use them to generate user and item embeddings. Then, given a user embedding you can find the nearest item embeddings and recommend these items, or cluster users so that they can be targeted with specific promotions. To generate such embeddings, first use the CREATE MODEL function to create a PCA, AutoEncoder or Matrix Factorization model and pass in your data as input, and then use ML.GENERATE_EMBEDDING function providing the model, and a table input to generate embeddings on this data. Getting started Support for multimodal embeddings and support for embeddings on structured data in BigQuery is now available in preview. Get started by following our documentation and tutorials. Have feedback? Let us know what you think at bqml-feedback@google.com. View the full article

Cloud SQL for SQL Server is a fully managed relational database that offers native SQL Server features along with several Google Cloud innovations, and allows migrating databases while retaining their existing versions or upgrading to newer ones. But migrating SQL Server databases to the cloud can be daunting. Concerns about downtime, complexity, and security risks can hinder your move to managed cloud databases. This week at Google Cloud Next, we announced the preview of support for SQL Server migrations to Cloud SQL for SQL Server in Database Migration Service , a fully managed serverless cloud service that performs database migrations with minimal downtime. Database Migration Service reduces migration complexity by loading an initial snapshot of the database followed by non-intrusive incremental load, reducing operational downtime. Let’s take a look at how it works and how to get started. Introducing SQL Server migration support Database Migration Service leverages built-in backup and restore to facilitate high-fidelity, low-downtime SQL Server migrations. Similar to other Database Migration Service homogeneous migration paths, SQL Server to Cloud SQL for SQL Server migrations are secure and reliable, and available at no additional charge. Database Migration Service offers: Minimal downtime and system overhead : Database Migration Service’s continuous native backup and restore technology helps ensure that your SQL Server source databases remain operational while the migration progresses, for reduced downtime. Because Database Migration Service leverages SQL Server backups to perform the migration, no additional overhead is introduced to your production operational systems. Serverless simplicity: As a managed service, you don’t need to manage any infrastructure for Database Migration Service, allowing your IT teams to focus on strategic initiatives rather than on managing complex migration setups. Security at the forefront: Database Migration Service supports encrypted SQL Server backups for robust protection throughout the migration. At Google Cloud, we’ve been working closely with customers who are looking to migrate SQL Server workloads. One of them is GoodRx Holdings, Inc., an American healthcare company that operates a telemedicine platform. "GoodRx is excited that Database Migration Service is now available to migrate SQL Server to Cloud SQL for SQL Server. Google Cloud's solutions help our business improve operational efficiencies in an intelligent, innovative, and cost-effective way." - Nitin Shingate, CTO GoodRx Migrating your SQL Server Databases with Database Migration Service Here’s how to start migrating your SQL Server workloads today using Database Migration Service: Navigate to the Database Migration area of the Google Cloud console, under Databases, and create a Connection Profile pointing to a Cloud Storage bucket to which your SQL Server backup files will be uploaded. Similarly, create another Connection Profile pointing to your chosen Cloud SQL for SQL Server instance. Create a migration job to connect both connection profiles and choose which databases you’d like to migrate. Test your migration job and make sure the test was successful as displayed below.,Then start it whenever you're ready. 5. Upload a full backup and continuously upload transaction log backups for each database to the Cloud Storage bucket at your desired frequency. 6. Once the full backups have been restored, the migration phase will change from “Initial Load” to “Incremental Load”. Database Migration Service keeps looking for new transaction log backup files and continuously restore them. 7. Monitor the progress of your migration jobs using the built-in metrics and logs to assess the right moment to complete the migration. 8. When ready to complete the migration, click “Promote” for Database Migration Service to recover the databases, making them available and completing the migration. Congratulations! You’ve completed your migration to Cloud SQL for SQL Server. Get started today Ready to migrate? Start your SQL Server migration to Google Cloud with the Database Migration Service. View the full article

Delta Lake is an open-source optimized storage layer that provides a foundation for tables in lake houses and brings reliability and performance improvements to existing data lakes. It sits on top of your data lake storage (like cloud object stores) and provides a performant and scalable metadata layer on top of data stored in the Parquet format. Organizations use BigQuery to manage and analyze all data types, structured and unstructured, with fine-grained access controls. In the past year, customer use of BigQuery to process multiformat, multicloud, and multimodal data using BigLake has grown over 60x. Support for open table formats gives you the flexibility to use existing open source and legacy tools while getting the benefits of an integrated data platform. This is enabled via BigLake — a storage engine that allows you to store data in open file formats on cloud object stores such as Google Cloud Storage, and run Google-Cloud-native and open-source query engines on it in a secure, governed, and performant manner. BigLake unifies data warehouses and lakes by providing an advanced, uniform data governance model. This week at Google Cloud Next '24, we announced that this support now extends to the Delta Lake format, enabling you to query Delta Lake tables stored in Cloud Storage or Amazon Web Services S3 directly from BigQuery, without having to export, copy, nor use manifest files to query the data. Why is this important? If you have existing dependencies on Delta Lake and prefer to continue utilizing Delta Lake, you can now leverage BigQuery native support. Google Cloud provides an integrated and price-performant experience for Delta Lake workloads, encompassing unified data management, centralized security, and robust governance. Many customers already harness the capabilities of Dataproc or Serverless Spark to manage Delta Lake tables on Cloud Storage. Now, BigQuery’s native Delta Lake support enables seamless delivery of data for downstream applications such as business intelligence, reporting, as well as integration with Vertex AI. This lets you do a number of things, including: Build a secure and governed lakehouse with BigLake’s fine-grained security model Securely exchange Delta Lake data using Analytics Hub Run data science workloads on Delta Lake using BigQuery ML and Vertex AI How to use Delta Lake with BigQuery Delta Lake tables follow the same table creation process as BigLake tables. Required roles To create a BigLake table, you need the following BigQuery identity and access management (IAM) permissions: bigquery.tables.create bigquery.connections.delegate Prerequisites Before you create a BigLake table, you need to have a dataset and a Cloud resource connection that can access Cloud Storage. Table creation using DDL Here is the DDL statement to create a Delta lake Table code_block <ListValue: [StructValue([('code', 'CREATE EXTERNAL TABLE `PROJECT_ID.DATASET.DELTALAKE_TABLE_NAME`\r\nWITH CONNECTION `PROJECT_ID.REGION.CONNECTION_ID`\r\nOPTIONS (\r\n format ="DELTA_LAKE",\r\n uris=[\'DELTA_TABLE_GCS_BASE_PATH\']);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e9803cb44f0>)])]> Querying Delta Lake tables After creating a Delta Lake BigLake table, you can query it using GoogleSQL syntax, the same as you would a standard BigQuery table. For example: code_block <ListValue: [StructValue([('code', 'SELECT FIELD1, FIELD2 FROM `PROJECT_ID.DATASET.DELTALAKE_TABLE_NAME`'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e9803cb4550>)])]> You can also enforce fine-grained security at the table level, including row-level and column-level security. For Delta Lake tables based on Cloud Storage, you can also use dynamic data masking. Conclusion We believe that BigQuery’s support for Delta Lake is a major step forward for customers building lakehouses using Delta Lake. This integration will make it easier for you to get insights from your data and make data-driven decisions. We are excited to see how you use Delta Lake and BigQuery together to solve their business challenges. For more information on how to use Delta Lake with BigQuery, please refer to the documentation. Acknowledgments: Mahesh Bogadi, Garrett Casto, Yuri Volobuev, Justin Levandoski, Gaurav Saxena, Manoj Gunti, Sami Akbay, Nic Smith and the rest of the BigQuery Engineering team. View the full article

Today, we are announcing the general availability of provider-defined functions in the AWS, Google Cloud, and Kubernetes providers in conjunction with the HashiCorp Terraform 1.8 launch. This release represents yet another step forward in our unique approach to ecosystem extensibility. Provider-defined functions will allow anyone in the Terraform community to build custom functions within providers and extend the capabilities of Terraform. Introducing provider-defined functions Previously, users relied on a handful of built-in functions in the Terraform configuration language to perform a variety of tasks, including numeric calculations, string manipulations, collection transformations, validations, and other operations. However, the Terraform community needed more capabilities than the built-in functions could offer. With the release of Terraform 1.8, providers can implement custom functions that you can call from the Terraform configuration. The schema for a function is defined within the provider's schema using the Terraform provider plugin framework. To use a function, declare the provider as a required_provider in the terraform{} block: terraform { required_version = ">= 1.8.0" required_providers { time = { source = "hashicorp/local" version = "2.5.1" } } }Provider-defined functions can perform multiple tasks, including: Transforming existing data Parsing combined data into individual, referenceable components Building combined data from individual components Simplifying validations and assertions To access a provider-defined function, reference the provider:: namespace with the local name of the Terraform Provider. For example, you can use the direxists function by including provider::local::direxists() in your Terraform configuration. Below you’ll find several examples of new provider-defined functions in the officially supported AWS, Google Cloud, and Kubernetes providers. Terraform AWS provider The 5.40 release of the Terraform AWS provider includes its first provider-defined functions to parse and build Amazon Resource Names (ARNs), simplifying Terraform configurations where ARN manipulation is required. The arn_parse provider-defined function is used to parse an ARN and return an object of individual referenceable components, such as a region or account identifier. For example, to get the AWS account ID from an Amazon Elastic Container Registry (ECR) repository, use the arn_parse function to retrieve the account ID and set it as an output: # create an ECR repository resource "aws_ecr_repository" "hashicups" { name = "hashicups" image_scanning_configuration { scan_on_push = true } } # output the account ID of the ECR repository output "hashicups_ecr_repository_account_id" { value = provider::aws::arn_parse(aws_ecr_repository.hashicups.arn).account_id } Running terraform apply against the above configuration outputs the AWS Account ID: Apply complete! Resources: 2 added, 0 changed, 0 destroyed. Outputs: hashicups_ecr_repository_account_id = "751192555662" Without the arn_parse function, you would need to define and test a combination of built-in Terraform functions to split the ARN and reference the proper index or define a regular expression to match on a substring. The function handles the parsing for you in a concise manner so that you do not have to worry about doing this yourself. The AWS provider also includes a new arn_build function that builds an ARN from individual attributes and returns it as a string. This provider-defined function can create an ARN that you cannot reference from another resource. For example, you may want to allow another account to pull images from your ECR repository. The arn_build function below constructs an ARN for an IAM policy using an account ID: # allow another account to pull from the ECR repository data "aws_iam_policy_document" "cross_account_pull_ecr" { statement { sid = "AllowCrossAccountPull" effect = "Allow" principals { type = "AWS" identifiers = [ provider::aws::arn_build("aws", "iam", "", var.cross_account_id, "root"), ] } actions = [ "ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer", ] } }The arn_build function helps to guide and simplify the process of combining substrings to form an ARN, and it improves readability compared to using string interpolation. Without it, you'd have to look up the exact ARN structure in the AWS documentation and manually test it. Terraform Google Cloud provider The 5.23 release of the Terraform Google Cloud provider adds a simplified way to get regions, zones, names, and projects from the IDs of resources that aren’t managed by your Terraform configuration. Provider-defined functions can now help parse Google IDs when adding an IAM binding to a resource that’s managed outside of Terraform: resource "google_cloud_run_service_iam_member" "example_run_invoker_jane" { member = "user:jane@example.com" role = "run.invoker" service = provider::google::name_from_id(var.example_cloud_run_service_id) location = provider::google::location_from_id(var.example_cloud_run_service_id) project = provider::google::project_from_id(var.example_cloud_run_service_id) }The Google Cloud provider also includes a new region_from_zone provider-defined function that helps obtain region names from a given zone (e.g. “us-west1” from “us-west1-a”). This simple string processing could be achieved in multiple ways using Terraform’s built-in functions previously, but the new function simplifies the process: locals { zone = “us-central1-a” # ways to derive the region “us-central1” using built-in functions region_1 = join("-", slice(split("-", local.zone), 0, 2)) region_2 = substr(local.zone, 0, length(local.zone)-2) # our new region_from_zone function makes this easier! region_3 = provider::google::region_from_zone(local.zone) }Terraform Kubernetes provider The 2.28 release of the Terraform Kubernetes provider includes provider-defined functions for encoding and decoding Kubernetes manifests into Terraform, making it easier for practitioners to work with the kubernetes_manifest resource. Users that have a Kubernetes manifest in YAML format can use the manifest_decode function to convert it into a Terraform object. The example below shows how to use the manifest_decode function by referring to a Kubernetes manifest in YAML format embedded in the Terraform configuration: locals { manifest = <If you prefer to decode a YAML file instead of using an embedded YAML format, you can do so by combining the built-in file function with the manifest_decode function. $ cat manifest.yaml --- kind: Namespace apiVersion: v1 metadata: name: test labels: name: testresource "kubernetes_manifest" "example" { manifest = provider::kubernetes::manifest_decode(file("${path.module}/manifest.yaml")) }If your manifest YAML contains multiple Kubernetes resources, you may use the manifestdecodemulti function to decode them into a list which can then be used with the for_each attribute on the kubernetes_manifest resource: $ cat manifest.yaml --- kind: Namespace apiVersion: v1 metadata: name: test-1 labels: name: test-1 --- kind: Namespace apiVersion: v1 metadata: name: test-2 labels: name: test-2 resource "kubernetes_manifest" "example" { for_each = { for m in provider::kubernetes::manifest_decode_multi(file("${path.module}/manifest.yaml"))): m.metadata.name => m } manifest = each.value }Getting started with provider-defined functions Provider-defined functions allow Terraform configurations to become more expressive and readable by declaring practitioner intent and reducing complex, repetitive expressions. To learn about all of the new launch-day provider-defined functions, please review the documentation and changelogs of the aforementioned providers: Terraform AWS provider Terraform Google provider Terraform Kubernetes provider Review our Terraform Plugin Framework documentation to learn more about how provider-defined functions work and how you can make your own. We are thankful to our partners and community members for their valuable contributions to the HashiCorp Terraform ecosystem. View the full article

Some Google Cloud customers will be able to run instances on the Arm-based Axion chip later this year. View the full article

Welcome to our live coverage of Google Cloud Next 2024! We're live in Las Vegas for a packed week of news and announcements from one of the world's biggest players when it comes to cloud innovation. The conference starts this morning with an opening keynote from Google Cloud CEO Thomas Kurian, and we've been promised plenty of other special guests. We'll be keeping you updated with all the latest announcements here, so stay tuned for everything you need to know for Google Cloud Next 2024! So what are we expecting to see from Google Cloud this week? In a word - AI. The company has been bullish in its use of AI across the board, and has been ramping up Gemini and Vertex in many of its product lines. Elsewhere, we're probably set to see a host of new updates and upgrades for Google Workspace - again, most likely to be around AI, but with the scale of the office software suite, who knows? We may also see some new hardware - with Nvidia recently announcing its latest flagship Blackwell chips, it's very likely we'll see something with Google announced here. Good morning from a beautifully sunny Las Vegas (although our view of the US eclipse yesterday was...sub-par, it has to be said) We're getting ready to head over for the day one keynote, hosted by none other than Google Cloud CEO Thomas Kurian himself. With a star-studded lineup of customers, partners and guests, the keynote should also include a whole host of news and announcements, so stay tuned for more soon! View the full article

We’re entering a new era for data analytics, going from narrow insights to enterprise-wide transformation through a virtuous cycle of data, analytics, and AI. At the same time, analytics and AI are becoming widely accessible, providing insights and recommendations to anyone with a question. Ultimately, we’re going beyond our own human limitations to leverage AI-based data agents to find deeply hidden insights for us. Organizations already recognize that data and AI can come together to unlock the value of AI for their business. Research from Google’s 2024 Data and AI Trends Report highlighted 84% of data leaders believe that generative AI will help their organization reduce time-to-insight, and 80% agree that the lines of data and AI are starting to blur. Today at Google Cloud Next ’24, we’re announcing new innovations for BigQuery and Looker that will help activate all of your data with AI: BigQuery is a unified AI-ready data platform with support for multimodal data, multiple serverless processing engines and built-in streaming and data governance to support the entire data-to-AI lifecycle. New BigQuery integrations with Gemini models in Vertex AI support multimodal analytics, vector embeddings, and fine-tuning of LLMs from within BigQuery, applied to your enterprise data. Gemini in BigQuery provides AI-powered experiences for data preparation, analysis and engineering, as well as intelligent recommenders to optimize your data workloads. Gemini in Looker enables business users to chat with their enterprise data and generate visualizations and reports—all powered by the Looker semantic data model that’s seamlessly integrated into Google Workspace. Let’s take a deeper look at each of these developments. BigQuery: the unified AI-ready data foundation BigQuery is now Google Cloud’s single integrated platform for data to AI workloads. BigLake, BigQuery’s unified storage engine, provides a single interface across BigQuery native and open formats for analytics and AI workloads, giving you the choice of where your data is stored and access to all of your data, whether structured or unstructured, along with a universal view of data supported by a single runtime metastore, built-in governance, and fine grained access controls. Today we’re expanding open format support with the preview of a fully managed experience for Iceberg, with DDL, DML and high throughput support. In addition to support for Iceberg and Hudi, we’re also extending BigLake capabilities with native support for the Delta file format, now in preview. “At HCA Healthcare we are committed to the care and improvement of human life. We are on a mission to redesign the way care is delivered, letting clinicians focus on patient care and using data and AI where it can best support doctors and nurses. We are building our unified data and AI foundation using Google Cloud's lakehouse stack, where BigQuery and BigLake enable us to securely discover and manage all data types and formats in a single platform to build the best possible experiences for our patients, doctors, and nurses. With our data in Google Cloud’s lakehouse stack, we’ve built a multimodal data foundation that will enable our data scientists, engineers, and analysts to rapidly innovate with AI." - Mangesh Patil, Chief Analytics Officer, HCA Healthcare We’re also extending our cross-cloud capabilities of BigQuery Omni. Through partnerships with leading organizations like Salesforce and our recent launch of bidirectional data sharing between BigQuery and Salesforce Data Cloud, customers can securely combine data across platforms with zero copy and zero ops to build AI models and predictions on combined Salesforce and BigQuery data. Customers can also enrich customer 360 profiles in Salesforce Data Cloud with data from BigQuery, driving additional personalization opportunities powered by data and AI. “It is great to collaborate without boundaries to unlock trapped data and deliver amazing customer experiences. This integration will help our joint customers tap into Salesforce Data Cloud's rich capabilities and use zero copy data sharing and Google AI connected to trusted enterprise data.” - Rahul Auradkar, EVP and General Manager of United Data Services & Einstein at Salesforce Building on this unified AI-ready data foundation, we are now making BigQuery Studio generally available, which already has hundreds of thousands of active users. BigQuery Studio provides a collaborative data workspace across data and AI that all data teams and practitioners can use to accelerate their data-to-AI workflows. BigQuery Studio provides the choice of SQL, Python, Spark or natural language directly within BigQuery, as well as new integrations for real-time streaming and governance. Customers’ use of serverless Apache Spark for data processing increased by over 500% in the past year1. Today, we are excited to announce the preview of our serverless engine for Apache Spark integrated within BigQuery Studio to help data teams work with Python as easily as they do with SQL, without having to manage infrastructure. The data team at Snap Inc. uses these new capabilities to converge toward a common data and AI platform with multiple engines that work across a single copy of data. This gives them the ability to enforce fine-grained governance and track lineage close to the data to easily expand analytics and AI use cases needed to drive transformation. To make data processing on real-time streams directly accessible from BigQuery, we’re announcing the preview of BigQuery continuous queries providing continuous SQL processing over data streams, enabling real-time pipelines with AI operators or reverse ETL. We are also announcing the preview of Apache Kafka for BigQuery as a managed service to enable streaming data workloads based on open-source APIs. We’re expanding our governance capabilities with Dataplex with new innovations for data-to-AI governance available in preview. You can now perform integrated search and drive gen AI-powered insights on your enterprise data, including data and models from Vertex AI, with a fully integrated catalog in BigQuery. We’re introducing column-level lineage in BigQuery and expanding lineage capabilities to support Vertex AI pipelines (available in preview soon) to help you better understand data-to-AI workloads. Finally, to facilitate governance for data-access at scale, we are launching governance rules in Dataplex. Multimodal analytics with new BigQuery and Vertex AI integrations With BigQuery’s direct integration with Vertex AI, we are now announcing the ability to connect models in Vertex AI with your enterprise data, without having to copy or move your data out of BigQuery. This enables multi-modal analytics using unstructured data, fine tuning of LLMs and the use of vector embeddings in BigQuery. Priceline, for instance, is using business data stored in BigQuery for LLMs across a wide range of applications. “BigQuery gave us a solid data foundation for AI. Our data was exactly where we needed it. We were able to connect millions of customer data points from hotel information, marketing content, and customer service chat and use our business data to ground LLMs.” - Allie Surina Dixon, Director of Data, Priceline The direct integration between BigQuery and Vertex AI now enables seamless preparation and analysis of multimodal data such as documents, audio and video files. BigQuery features rich support for analyzing unstructured data using object tables and Vertex AI Vision, Document AI and Speech-to-Text APIs. We are now enabling BigQuery to analyze images and video using Gemini 1.0 Pro Vision, making it easier than ever to combine structured with unstructured data in data pipelines using the generative AI capabilities of the latest Gemini models. BigQuery makes it easier than ever to execute AI on enterprise data by providing the ability to build prompts based on your BigQuery data, and use of LLMs for sentiment extraction, classification, topic detection, translation, classification, data enrichment and more. BigQuery now also supports generating vector embeddings and indexing them at scale using vector and semantic search. This enables new use cases that require similarity search, recommendations or retrieval of your BigQuery data, including documents, images or videos. Customers can use the semantic search in the BigQuery SQL interface or via our integration with gen AI frameworks such as LangChain and leverage Retrieval Augmented Generation based on their enterprise data. Gemini in BigQuery and Gemini in Looker for AI-powered assistance Gen AI is creating new opportunities for rich data-driven experiences that enable business users to ask questions, build custom visualizations and reports, and surface new insights using natural language. In addition to business users, gen AI assistive and agent capabilities can also accelerate the work of data teams, spanning data exploration, analysis, governance, and optimization. In fact, more than 90% of organizations believe business intelligence and data analytics will change significantly due to AI. Today, we are announcing the public preview of Gemini in BigQuery, which provides AI-powered features that enhance user productivity and optimize costs throughout the analytics lifecycle, from ingestion and pipeline creation to deriving valuable insights. What makes Gemini in BigQuery unique is its contextual awareness of your business through access to metadata, usage data, and semantics. Gemini in BigQuery also goes beyond chat assistance to include new visual experiences such as data canvas, a new natural language-based experience for data exploration, curation, wrangling, analysis, and visualization workflows. Imagine you are a data analyst at a bikeshare company. You can use the new data canvas of Gemini in BigQuery to explore the datasets, identify the top trips and create a customized visualization, all using natural language prompts within the same interface Gemini in BigQuery capabilities extend to query recommendations, semantic search capabilities, low-code visual data pipeline development tools, and AI-powered recommendations for query performance improvement, error minimization, and cost optimization. Additionally, it allows users to create SQL or Python code using natural language prompts and get real-time suggestions while composing queries. Today, we are also announcing the private preview of Gemini in Looker to enable business users and analysts to chat with their business data. Gemini in Looker capabilities include conversational analytics, report and formula generation, LookML and visualization assistance, and automated Google slide generation. What’s more, these capabilities are being integrated with Workspace to enable users to easily access beautiful data visualizations and insights right where they work. Imagine you’re an ecommerce store. You can query Gemini in Looker to learn sales trends and market details and immediately explore the insights, with details on how the charts were created. To learn more about our data analytics product innovations, hear customer stories, and gain hands-on knowledge from our developer experts, join our data analytics spotlights and breakout sessions at Google Cloud Next ‘24, or watch them on-demand. 1. Google internal data - YoY growth of data processed using Apache Spark on Google Cloud compared with Feb ‘23 View the full article

Protecting people, data, and critical assets is a crucial responsibility for modern organizations, yet conventional security approaches often struggle to address the escalating velocity, breadth, and intricacy of modern cyberattacks. Bolting on more new security products is simply not a viable long-term strategy. What organizations need from their security solutions is a convergence of essential capabilities that brings simplicity, streamlines operations, and enhances efficiency and effectiveness. Today at Google Cloud Next, we are announcing innovations across our security portfolio that are designed to deliver stronger security outcomes and enable every organization to make Google a part of their security team. Increasing speed and productivity with Gemini in Security Generative AI offers tremendous potential to tip the balance in favor of defenders and we continue to infuse AI-driven capabilities into our products. Today we’re announcing the following new AI capabilities: Gemini in Security Operations is coming to the entire investigation lifecycle, building on our December GA of natural language search and case summaries in Chronicle. A new assisted investigation feature, generally available at the end of this month, will guide analysts through their workflow wherever they are in Chronicle Enterprise and Chronicle Enterprise Plus. Gemini recommends actions based on the context of an investigation, and can run searches and create detection rules to improve response times. Plus, analysts can now ask Gemini for the latest threat intelligence from Mandiant directly in-line — including any indicators of compromise found in their environment — and Gemini will navigate users to the most relevant pages in the integrated platform for deeper investigation. Gemini in Security Operations allows users to quickly investigate incidents and alerts using conversational chat in Chronicle. “Gemini in Security Operations is enabling us to enhance the efficiency of our Cybersecurity Operations Center program as we continue to drive operational excellence,” said Ronald Smalley, senior vice president of Cybersecurity Operations, Fiserv. “Detection engineers can create detections and playbooks with less effort, and security analysts can find answers quickly with intelligent summarization and natural language search. This is critical as SOC teams continue to manage increasing data volumes and need to detect, validate, and respond to events faster.“ Gemini in Threat Intelligence now offers conversational search across Mandiant’s vast and growing repository of threat intelligence directly from frontline investigations — a grounded experience, now in preview. Plus, VirusTotal now automatically ingests OSINT reports, which Gemini summarizes directly in the platform — a feature that’s generally available now. Gemini in Security Command Center now offers preview features that let security teams search for threats and other security events using natural language. It can also provide summaries of critical- and high-priority misconfiguration and vulnerability alerts, and summarize attack paths to help understand cloud risks for remediation. We are also infusing AI in many of our cloud platform’s security services. Today, we’re announcing previews of new capabilities in Gemini Cloud Assist, including: IAM Recommendations, which can provide straightforward, contextual recommendations to remove roles from over-permissioned users or service accounts to help uplevel IAM posture and reduce risk exposure. Key Insights, which can provide assistance during encryption key creation based on its understanding of your data, your encryption preferences, and your compliance needs. Confidential Computing Insights, which can recommend options for adding confidential computing protection for your most sensitive workloads based on your data and your compute usage. Delivering a new frontline of defense for the enterprise Chrome Enterprise Premium is a new offering that redefines, simplifies, and strengthens endpoint security. It brings together the most popular and trusted enterprise browser with Google’s advanced security capabilities, including threat and data protection, Zero Trust access controls, enterprise policy controls, and security insights and reporting. With Chrome Enterprise Premium, which is generally available today, hundreds of millions of enterprise users can get additional protection delivered instantly where they do their work every day. "With Chrome Enterprise Premium, we have confidence in Google’s security expertise, including Project Zero’s cutting-edge security research and fast security patches. We set up data loss prevention restrictions and warnings for sharing sensitive information in applications like Generative AI platforms and noticed a noteworthy 50% reduction in content transfers,” said Nick Reva, head of corporate security engineering, Snap Inc. Turning intelligence into action Our focus on intelligence-driven outcomes continues with the launch of Applied Threat Intelligence in Google Security Operations. Applied threat intelligence takes our industry-leading global threat visibility and automatically applies it to each customer’s unique environment. It can help security operations teams uncover more threats with less effort and use the most up-to-date threat intelligence to address them before they create damage or loss. Managing cloud risk Security Command Center Enterprise, the industry’s first cloud risk-management solution that fuses proactive cloud security and enterprise security operations, is now generally available. This new solution offers security teams a single view of their posture controls, active threats, cloud identities, data, and more, while integrating remediation and issue accountability into end-to-end workflows. Mandiant Hunt for Security Command Center Enterprise is now in preview, and offers on-demand human expertise that can become an extension of internal security operations teams. Hundreds of elite-level analysts and researchers are available on-call to proactively find elusive threats in organizations’ SCC environments. New security capabilities in our trusted cloud We continue our regular delivery of new security controls and capabilities on our cloud platform to help organizations meet evolving policy, compliance, and business objectives. Today we’re announcing the following updates: For Identity and Access Management: Privileged Access Manager (PAM), now available in preview, is designed to help mitigate risks tied to privileged access misuse or abuse. PAM can help customers shift from always-on standing privileges towards on-demand access with just-in-time, time-bound, and approval-based access elevations. Principal Access Boundary (PAB) is a new, identity-centric control now in preview. It can empower security administrators to enforce restrictions on IAM principals so that they can only access authorized resources within a specific defined boundary. For Network Security: Cloud NGFW Enterprise is now generally available. Our cloud-first next generation firewall (NGFW) includes threat protection powered by Palo Alto Networks with a unique distributed architecture that can provide granular control at the workload level. Cloud Armor Enterprise, now generally available, offers a pay-as-you-go model that includes advanced network DDoS protection, web application firewall capabilities, network edge policy, adaptive protection, and threat intelligence to help protect your cloud applications and services. For Data Security: Confidential Accelerators: Confidential VMs on Intel TDX are now in preview and available on the C3 machine series with Intel TDX. For AI and ML workloads, we support Intel AMX, which provides CPU-based acceleration by default on C3 series Confidential VMs. In addition, Confidential Compute will also be coming to A3 VMs with NVIDIA H100 GPUs in preview later this year. With these announcements, our Confidential Computing portfolio now spans Intel, AMD, and NVIDIA hardware. Sensitive Data Protection integration with Cloud SQL is now generally available, and is deeply integrated into the Security Command Center Enterprise risk engine. This powerful combination can pinpoint high-value assets, analyze vulnerabilities in databases, and simulate real-world attack scenarios that can enable you to proactively address risks and safeguard data. Key management with Autokey is now in preview. Autokey simplifies creating and managing customer encryption keys (CMEK) by ensuring you use the right key type for each resource, thus reducing complexity and management overhead. Plus, Autokey can help you adhere to industry best practices for data security. Expanded regions available for bare metal HSM deployments allows you to deploy your own HSMs in PCI-compliant facilities with your Google Cloud workloads. For our Regulated Cloud offerings: Regional Controls for Assured Workloads is now in preview and is available in 32 cloud regions in 14 countries. The Regional Controls package can enforce data residency for customer content at rest, offers administrative access transparency, as well as compliant service restriction and monitoring. Regional controls are available at no additional cost. Audit Manager is now in preview. Audit Manager can help customers drastically simplify their compliance audit process by automating control verification with proof of compliance for their workloads and data on Google Cloud. Take your next security steps with Google Cloud Google’s decade of AI innovation, coupled with our security expertise, means we are strongly positioned to help you protect your users and brand by becoming an integral part of your security team. For more on our Next ‘24 announcements, you can watch our security spotlight, and check out the many great security breakout sessions at Google Cloud Next — live or on-demand. View the full article

Multi-cluster Ingress (MCI) is an advanced feature typically used in cloud computing environments that enables the management of ingress (the entry point for external traffic into a network) across multiple Kubernetes clusters. This functionality is especially useful for applications that are deployed globally across several regions or clusters, offering a unified method to manage access to these applications. MCI simplifies the process of routing external traffic to the appropriate cluster, enhancing both the reliability and scalability of applications. Here are key features and benefits of Multi-cluster Ingress: Global Load Balancing: MCI can intelligently route traffic to different clusters based on factors like region, latency, and health of the service. This ensures users are directed to the nearest or best-performing cluster, improving the overall user experience. Centralized Management: It allows for the configuration of ingress rules from a single point, even though these rules are applied across multiple clusters. This simplification reduces the complexity of managing global applications. High Availability and Redundancy: By spreading resources across multiple clusters, MCI enhances the availability and fault tolerance of applications. If one cluster goes down, traffic can be automatically rerouted to another healthy cluster. Cross-Region Failover: In the event of a regional outage or a significant drop in performance within one cluster, MCI can perform automatic failover to another cluster in a different region, ensuring continuous availability of services. Cost Efficiency: MCI helps optimize resource utilization across clusters, potentially leading to cost savings. Traffic can be routed to clusters where resources are less expensive or more abundant. Simplified DNS Management: Typically, MCI solutions offer integrated DNS management, automatically updating DNS records based on the health and location of clusters. This removes the need for manual DNS management in a multi-cluster setup. How What is Multi-cluster Ingress (MCI) works? Multi-cluster Ingress (MCI) works by managing and routing external traffic into applications running across multiple Kubernetes clusters. This process involves several components and steps to ensure that traffic is efficiently and securely routed to the appropriate destination based on predefined rules and policies. Here’s a high-level overview of how MCI operates: 1. Deployment Across Multiple Clusters Clusters Preparation: You deploy your application across multiple Kubernetes clusters, often spread across different geographical locations or cloud regions, to ensure high availability and resilience. Ingress Configuration: Each cluster has its own set of resources and services that need to be exposed externally. With MCI, you configure ingress resources that are aware of the multi-cluster environment. 2. Central Management and Configuration Unified Ingress Control: A central control plane is used to manage the ingress resources across all participating clusters. This is where you define the rules for how external traffic should be routed to your services. DNS and Global Load Balancer Setup: MCI integrates with global load balancers and DNS systems to direct users to the closest or most appropriate cluster based on various criteria like location, latency, and the health of the clusters. 3. Traffic Routing Initial Request: When a user makes a request to access the application, the DNS resolution directs the request to the global load balancer. Global Load Balancing: The global load balancer evaluates the request against the configured routing rules and the current state of the clusters (e.g., load, health). It then selects the optimal cluster to handle the request. Cluster Selection: The criteria for cluster selection can include geographic proximity to the user, the health and capacity of the clusters, and other custom rules defined in the MCI configuration. Request Forwarding: Once the optimal cluster is selected, the global load balancer forwards the request to an ingress controller in that cluster. Service Routing: The ingress controller within the chosen cluster then routes the request to the appropriate service based on the path, host, or other headers in the request. 4. Health Checks and Failover Continuous Monitoring: MCI continuously monitors the health and performance of all clusters and their services. This includes performing health checks and monitoring metrics to ensure each service is functioning correctly. Failover and Redundancy: In case a cluster becomes unhealthy or is unable to handle additional traffic, MCI automatically reroutes traffic to another healthy cluster, ensuring uninterrupted access to the application. 5. Scalability and Maintenance Dynamic Scaling: As traffic patterns change or as clusters are added or removed, MCI dynamically adjusts routing rules and load balancing to optimize performance and resource utilization. Configuration Updates: Changes to the application or its deployment across clusters can be managed centrally through the MCI configuration, simplifying updates and maintenance. Example Deployment YAML for Multi-cluster Ingress with FrontendConfig and BackendConfig This example includes: A simple web application deployment. A service to expose the application within the cluster. A MultiClusterService to expose the service across clusters. A MultiClusterIngress to expose the service externally with FrontendConfig and BackendConfig. The post What is Multi-cluster Ingress (MCI) appeared first on DevOpsSchool.com. View the full article

Google Cloud Next ‘24 is right around the corner, and this year, there will be even more for developers to enjoy. Whether you're on the generative AI kick or keeping up with what's new with Javascript frameworks, there's a session for technical practitioners in the Application Developers Zone! Meet the experts and get inspired to solve your biggest technical challenges over three days in Las Vegas, April 9-11. With over 600 sessions available, you can choose from a wide range of topics, covering everything from security to growing revenue with AI. Here are a few of the sessions you won't want to miss: 1. Building generative AI apps on Google Cloud with LangChain Learn how to use LangChain, the most popular open-source framework for building LLM-based apps, on Cloud Run to add gen AI features to your own applications. See how to combine it with Cloud SQL's pgvector extension for vector storage to quickly locate similar data points with vector search and reduce the cost of deploying models with Vertex Endpoints. 2. How to deploy all the JavaScript frameworks to Cloud Run Join this session, where we’ll deploy as many JavaScript frameworks as we can, including Angular, React, and Node.js, as quickly as we can to prove that it can be done and have some fun! 3. Build full stack applications using Firebase & Google Cloud Discover how Firebase and Google Cloud simplify full-stack development, empowering you to rapidly build scalable, enterprise-grade applications. 4. Get developer assistance customized to your organization code with Gemini Join Gemini product managers and Capgemini to learn how to customize Duet AI with your own private codebase and bring AI code assistance to your development teams. Also be sure to check out the Innovators Hive — a central hub where you can discover what’s new and next at Google Cloud — to dive into interactive demos and talk face-to-face with our developer experts. Here are a few of the demos you can expect to find in the Innovators Hive AppDev zone: Write your own Cloud Functions (with the help of Duet AI/Gemini) to interact with our programmable pinball machine that uses Pub/Sub to collect events from the game in real time. Learn all about serving a Gemma large language model using graphical processing units (GPUs) on Google Kubernetes Engine (GKE) Autopilot, working with the 2B and 7B parameter models in particular. There’s so much more to enjoy at Next ‘24 that you’ll have to see it to believe it. And if you can’t go in person, you can still register for a digital pass, and access keynotes, 100+ breakout sessions on demand. View the full article

The rise of data collaboration and use of external data sources highlights the need for robust privacy and compliance measures. In this evolving data ecosystem, businesses are turning to clean rooms to share data in low-trust environments. Clean rooms enable secure analysis of sensitive data assets, allowing organizations to unlock insights without compromising on privacy. To facilitate this type of data collaboration, we launched the preview of data clean rooms last year. Today, we are excited to announce that BigQuery data clean rooms is now generally available. Backed by BigQuery, customers can now share data in place with analysis rules to protect the underlying data. This launch includes a streamlined data contributor and subscriber experience in the Google Cloud console, as well as highly requested capabilities such as: Join restrictions: Limits the joins that can be on specific columns for data shared in a clean room, preventing unintended or unauthorized connections between data. Differential privacy analysis rule: Enforces that all queries on your shared data use differential privacy with the parameters that you specify. The privacy budget that you specify also prevents further queries on that data when the budget is exhausted. List overlap analysis rule: Restricts the output to only display the intersecting rows between two or more views joined in a query. Usage metrics on views: Data owners or contributors see aggregated metrics on the views and tables shared in a clean room. Using data clean rooms in BigQuery does not require creating copies of or moving sensitive data. Instead, the data can be shared directly from your BigQuery project and you remain in full control. Any updates you make to your shared data are reflected in the clean room in real-time, ensuring everyone is working with the most current data. Create and deploy clean rooms in BigQuery BigQuery data clean rooms are available in all BigQuery regions. You can set up a clean room environment using the Google Cloud console or using APIs. During this process, you set permissions and invite collaborators within or outside organizational boundaries to contribute or subscribe to the data. Enforce analysis rules to protect underlying data When sharing data into a clean room, you can configure analysis rules to protect the underlying data and determine how the data can be analyzed. BigQuery data clean rooms support multiple analysis rules including aggregation, differential privacy, list overlap, and join restrictions. The new user experience within Cloud console lets data contributors configure these rules without needing to use SQL. Lastly, by default, a clean room employs restricted egress to prevent subscribers from exporting or copying the underlying data. However, data contributors can choose to allow the export and copying of query results for specific use cases, such as activation. Monitor usage and stay in control of your data The data owner or contributor is always in control of their respective data in a clean room. At any time, a data contributor can revoke access to their data. Additionally, as the clean room owner, you can adjust access using subscription management or privacy budgets to prevent subscribers from performing further analysis. Additionally, data contributors receive aggregated logs and metrics, giving them insights into how their data is being used within the clean room. This promotes both transparency and a clearer understanding of the collaborative process. What BigQuery data clean room customers are saying Customers across all industries are already seeing tremendous success with BigQuery data clean rooms. Here’s what some of our early adopters and partners had to say: “With BigQuery data clean rooms, we are now able to share and monetize more impactful data with our partners while maintaining our customers' and strategic data protection.” - Guillaume Blaquiere, Group Data Architect, Carrefour “Data clean rooms in BigQuery is a real accelerator for L'Oréal to be able to share, consume, and manage data in a secure and sustainable way with our partners.” - Antoine Castex, Enterprise Data Architect, L’Oréal “BigQuery data clean rooms equip marketing teams with a powerful tool for advancing privacy-focused data collaboration and advanced analytics in the face of growing signal loss. LiveRamp and Habu, which independently were each early partners of BigQuery data clean rooms, are excited to build on top of this foundation with our combined interoperable solutions: a powerful application layer, powered by Habu, accelerates the speed to value for technical and business users alike, while cloud-native identity, powered by RampID in Google Cloud, maximizes data fidelity and ecosystem connectivity for all collaborators. With BigQuery data clean rooms, enterprises will be empowered to drive more strategic decisions with actionable, data-driven insights.” - Roopak Gupta, VP of Engineering, LiveRamp “In today’s marketing landscape, where resources are limited and the ecosystem is fragmented, solutions like the data clean room we are building with Google Cloud can help reduce friction for our clients. This collaborative clean room ensures privacy and security while allowing Stagwell to integrate our proprietary data to create custom audiences across our product and service offerings in the Stagwell Marketing Cloud. With the continued partnership of Google Cloud, we can offer our clients integrated Media Studio solutions that connect brands with relevant audiences, improving customer journeys and making media spend more efficient.” - Mansoor Basha, Chief Technology Officer, Stagwell Marketing Cloud “We are extremely excited about the General Availability announcement of BigQuery data clean rooms. It's been great collaborating with Google Cloud on this initiative and it is great to see it come to market.. This release enables production-grade secure data collaboration for the media and advertising industry, unlocking more interoperable planning, activation and measurement use cases for our ecosystem.” - Bosko Milekic, Chief Product Officer, Optable Next steps Whether you're an advertiser trying to optimize your advertising effectiveness with a publisher, or a retailer improving your promotional strategy with a CPG, BigQuery data clean rooms can help. Get started today by using this guide, starting a free trial with BigQuery, or contacting the Google Cloud sales team. View the full article

We are excited to announce that differential privacy enforcement with privacy budgeting is now available in BigQuery data clean rooms to help organizations prevent data from being reidentified when it is shared. Differential privacy is an anonymization technique that limits the personal information that is revealed in a query output. Differential privacy is considered to be one of the strongest privacy protections that exists today because it: is provably private supports multiple differentially private queries on the same dataset can be applied to many data types Differential privacy is used by advertisers, healthcare companies, and education companies to perform analysis without exposing individual records. It is also used by public sector organizations that comply with the General Data Protection Regulation (GDPR), the Health Insurance Portability and Accountability Act (HIPAA), the Family Educational Rights and Privacy Act (FERPA), and the California Consumer Privacy Act (CCPA). What can I do with differential privacy? With differential privacy, you can: protect individual records from re-identification without moving or copying your data protect against privacy leak and re-identification use one of the anonymization standards most favored by regulators BigQuery customers can use differential privacy to: share data in BigQuery data clean rooms while preserving privacy anonymize query results on AWS and Azure data with BigQuery Omni share anonymized results with Apache Spark stored procedures and Dataform pipelines so they can be consumed by other applications enhance differential privacy implementations with technology from Google Cloud partners Gretel.ai and Tumult Analytics call frameworks like PipelineDP.io So what is BigQuery differential privacy exactly? BigQuery differential privacy is three capabilities: Differential privacy in GoogleSQL – You can use differential privacy aggregate functions directly in GoogleSQL Differential privacy enforcement in BigQuery data clean rooms – You can apply a differential privacy analysis rule to enforce that all queries on your shared data use differential privacy in GoogleSQL with the parameters that you specify Parameter-driven privacy budgeting in BigQuery data clean rooms – When you apply a differential privacy analysis rule, you also set a privacy budget to limit the data that is revealed when your shared data is queried. BigQuery uses parameter-driven privacy budgeting to give you more granular control over your data than query thresholds do and to prevent further queries on that data when the budget is exhausted. BigQuery differential privacy enforcement in action Here’s how to enable the differential privacy analysis rule and configure a privacy budget when you add data to a BigQuery data clean room. Subscribers of that clean room must then use differential privacy to query your shared data. Subscribers of that clean room cannot query your shared data once the privacy budget is exhausted. Get started with BigQuery differential privacy BigQuery differential privacy is configured when a data owner or contributor shares data in a BigQuery data clean room. A data owner or contributor can share data using any compute pricing model and does not incur compute charges when a subscriber queries that data. Subscribers of a data clean room incur compute charges when querying shared data that is protected with a differential privacy analysis rule. Those subscribers are required to use on-demand pricing (charged per TB) or the Enterprise Plus edition (charged per slot hour). Create a clean room where all queries are protected with differential privacy today and let us know where you need help. Related Article Privacy-preserving data sharing now generally available with BigQuery data clean rooms Now GA, BigQuery data clean rooms has a new data contributor and subscriber experience, join restrictions, new analysis rules, usage metr... Read Article View the full article

Building all the tools you need from scratch or having complex installments for a cloud environment is a thing of the past now. In this fast-paced world, we need solutions that save time and make us more efficient. That’s where a one-stop for discovering, deploying, and managing all the essential tools you need comes into […]View the full article

Experience the magic at Google Cloud Next '24, where access to new opportunities awaits, helping you create, innovate, and supercharge your startup's growth. In the Startup Lounge, you’ll be able to grow your network, receive practical training on launching and scaling more efficiently, and acquire essential strategies to propel your startup forward. Over three days, dive into exclusive startup sessions and founder stories tailored to a startup's practical needs and transformational goals. You’ll also have the chance to learn the secrets to securing funding from Menlo Ventures, understand why AI unicorns are increasingly choosing Google Cloud and find out about the state of the startup landscape around the world. This year’s agenda is packed with exciting moments you won’t want to miss. Come listen to Dario Amodei, CEO and Co-Founder of Anthropic, and Elad Gil, a renowned entrepreneur and investor, as they discuss the transformative impact of generative AI on the business landscape in a fireside chat. You’ll also want to bookmark sessions from the Founder’s Series, which will offer insights on redefining strategies related to cost, productivity, and innovation from the founders of startups like LangChain, Hugging Face, Runway, Character.AI, AI21 Labs, Contextual AI, and more. These are only some of the can’t-miss sessions geared towards startup leaders — make sure you explore the full list here. But that’s not all, here’s what else startups can look forward to at Next ‘24. Build meaningful connections in the Startup Lounge Make sure to join us after the Main Keynote for the "Startups at Next” Kickoff Event, where we’ll be sharing some big announcements that will create new opportunities for you and your startup to grow with AI. Connect with peers, venture capitalists, and industry leaders during events such as the Xoogler Networking Event, the DEI Founder’s Ice Cream Social, and the Founder's Toast. Access tailored expertise with personal Office Hours Facing challenges or seeking to refine your strategy? Book direct access to Google experts and engineers in 15-minute Startup Expert sessions or pre-book 30-minute Office Hours sessions. These opportunities allow you to address technical questions, strategize on business development, or delve into AI, equipping you with the tools to unlock your startup's full potential. Book today before they are gone! Experience innovation at the Startup Showcases The excitement at Google Cloud Next '24 goes beyond traditional formats with the Startup Lounge featuring unique activations that foster creativity and growth. Witness over 25 startups launching new products or features on stage, enjoy a life-sized game of Chutes & Ladders by Product Science, and explore the future with Octo.ai in the Google Next Yearbook. Engage with AssemblyAI Speech-to-text AI for a unique fortune-telling experience and participate in the Suggestic three-day wellness challenge as you navigate Next ‘24. Don't miss out on grand prizes from each showcase and more. Apply for Cloud Credits and More with 1:1 Application Help This year, dedicated staff will be available at Next '24 to guide you through the Google for Startups Cloud Program application process. By joining, you'll gain access to startup experts, coverage for your Google Cloud and Firebase costs up to $200,000 USD (up to $350,000 USD for AI startups) over two years, technical training, business support, and exclusive Google-wide offers. We hope to see you at the event, and look forward to welcoming startup leaders from around the globe to Mandalay Bay in Las Vegas. View the full article

Google Kubernetes Engine (GKE) offers two different ways to perform service discovery and DNS resolution: the in-cluster kube-dns functionality, and GCP managed Cloud DNS. Either approach can be combined with the performance-enhancing NodeLocal DNSCache add-on. New GKE Autopilot clusters use Cloud DNS as a fully managed DNS solution for your GKE Autopilot clusters without any configuration required on your part. But for GKE Standard clusters, you have the following DNS provider choices: Kube-dns (default) Cloud DNS - configured for either cluster-scope or VPC scope, and Install and run your own DNS (like Core DNS) In this blog, we break down the differences between the DNS providers for your GKE Standard clusters, and guide you to the best solution for your specific situation. Kube-DNS kube-dns is the default DNS provider for Standard GKE clusters, providing DNS resolution for services and pods within the cluster. If you select this option, GKE deploys the necessary kube-dns components such as Kube-dns pods, Kube-dns-autoscaler, Kube-dns configmap and Kube-dns service in the kube-system namespace. kube-dns is the default DNS provider for GKE Standard clusters and the only DNS provider for Autopilot clusters running versions earlier than 1.25.9-gke.400 and 1.26.4-gke.500. Kube-dns is a suitable solution for workloads with moderate DNS query volumes that have stringent DNS resolution latency requirements (e.g. under ~2-4ms). Kube-dns is able to provide low latency DNS resolution for all DNS queries as all the DNS resolutions are performed within the cluster. If you notice DNS timeouts or failed DNS resolutions for bursty workload traffic patterns when using kube-dns, consider scaling the number of kube-dns pods, and enabling NodeLocal DNS cache for the cluster. You can scale the number of kube-dns pods beforehand using Kube-dns autoscaler, and manually tuning it to the cluster's DNS traffic patterns. Using kube-dns along with Nodelocal DNS cache (discussed below) also reduces overhead on the kube-dns pods for DNS resolution of external services. While scaling up kube-dns and using NodeLocal DNS Cache(NLD) helps in the short term, it does not guarantee reliable DNS resolution during sudden traffic spikes. Hence migrating to Cloud DNS provides a more robust and long-term solution for improved reliability of DNS resolution consistently across varying DNS query volumes. You can update the DNS provider for your existing GKE Standard from kube-dns to Cloud DNS without requiring to re-create your existing cluster. For logging the DNS queries when using kube-dns, there is manual effort required in creating a new kube-dns debug pod with log-queries enabled. Cloud DNS Cloud DNS is a Google-managed service that is designed for high scalability and availability. In addition, Cloud DNS elastically scales to adapt to your DNS query volume, providing consistent and reliable DNS query resolution regardless of traffic volume. Cloud DNS simplifies your operations and minimizes operational overhead since it is a Google managed service and does not require you to maintain any additional infrastructure. Cloud DNS supports dns resolutions across the entire VPC, which is something not currently possible with kube-dns. Also, while using Multi Cluster Services (MCS) in GKE, Cloud DNS provides DNS resolution for services across your fleet of clusters. Unlike kube-dns, Google Cloud’s hosted DNS service Cloud DNS provides Pod and Service DNS resolution that auto-scales and offers a 100% service-level agreement, reducing DNS timeouts and providing consistent DNS resolution latency for heavy DNS workloads. Cloud DNS also integrates with Cloud Monitoring, giving you greater visibility into DNS queries for enhanced troubleshooting and analysis. The Cloud DNS controller automatically provisions DNS records for pods and services in Cloud DNS for ClusterIP, headless and external name services. You can configure Cloud DNS to provide GKE DNS resolution in either VPC or Cluster (the default) scope. With VPC scope, the DNS records are resolvable with the entire VPC. This is achieved with the private DNS zone that gets created automatically. With Cluster scope, the DNS records are resolvable only within the cluster. While Cloud DNS offers enhanced features, it does come with usage-based costs. You save on compute costs and overhead by removing kube-dns pods when using Cloud DNS. Considering the typical cluster size workload traffic patterns, Cloud DNS is usually more cost effective than running kube-dns You can migrate clusters from kube-dns to Cloud DNS cluster scope without downtime or changes to your applications. The reverse (migrating from Cloud DNS to kube-dns) is not a seamless operation. NodeLocal DNSCache NodeLocal DNSCache is a GKE add-on that you can run in addition to kube-dns and Cloud DNS. The node-local-dns pod gets deployed on the GKE nodes after the option has been enabled (subject to a node upgrade procedure). Nodelocal DNS Cache (NLD) helps to reduce the average DNS resolution times by resolving the DNS requests locally on the same nodes as the pods, and only forwards requests that it cannot resolve to the other DNS servers in the cluster. This is a great fit for clusters that have heavy internal DNS query loads. Enable NLD during maintenance windows. Please note that node pools must be re-created for this change to take effect. Final thoughts The choice of DNS provider for your GKE Standard cluster has implications for the performance and reliability, in addition to your operations and overall service discovery architecture. Hence, it is crucial for GKE Standard users to understand their DNS options taking into account their application and architecture objectives. Standard GKE clusters allow you to use either kube-dns or Cloud DNS as your DNS provider, allowing you to optimize for either low latency DNS resolution or a simple, scalable and reliable DNS solution for GKE Standard clusters. You can learn more about DNS for your GKE cluster from the GKE documentation . If you have any further questions, feel free to contact us. We thank the Google Cloud team member who contributed to the blog: Selin Goksu, Technical Solutions Developer, Google View the full article

We're excited to announce a major enhancement to Google Cloud Backup and DR, making it simpler than ever to safeguard your critical Google Compute Engine virtual machines (VMs). You can now leverage the power of Google Cloud tags, including inheritance, to easily configure backup policies for Compute Engine VMs, ensuring consistent protection of your dynamic cloud environments. The challenge of managing VM backups Managing backups for a large number of Compute Engine VMs can be complex, especially when VMs are frequently added, removed, or modified. Manually assigning backup policies to individual VMs is time-consuming and can be error-prone, potentially leaving vital resources unprotected. The power of tag-based backups With our new tag-based backup approach, you can leverage the organizational power of tags to automate the protection of your Compute Engine VMs. Here's how it works: Assign your tags: Apply meaningful tags to your organization, folders, projects, or directly to your VMs. These tags might reflect application names, environments (production, development, testing), or criticality levels. Create tag-based policies: Within Google Backup and DR, you can create backup policies with your desired backup schedules, retention periods, and recovery regions. After policy creation, you can assign them to specific tags. Automate protection: Any VM with a matching tag is automatically assigned the backup policy. New VMs inheriting those tags immediately gain protection. Benefits of tag-based backups Simplified management: Drastically reduce the administrative overhead of configuring VM backups at scale by updating the policy attached to the tag rather than at an individual VM backup level. Consistent protection: Ensure new VMs with inherited tags from the project, folder, or organization are automatically protected without manual intervention. Flexibility: Adjust your backup strategies easily by modifying tags or creating new tag-based policies, including support for tags assigned using Terraform For example, you can easily tag all production projects with the tag backupdr-dynamicprotect:production. You could then create a backup policy that does the following: Takes daily snapshots of VMs in those projects that inherit the backupdr-dynamicprotect:production tag Retains backups for 30 days Updates your Terraform script to include your new protection tag to ensure protection of every new Compute Engine instance that’s created Getting started Review the Google Backup and DR documentation on tag-based backups. Develop a tagging strategy for your Compute Engine VMs. Create backup policies that target your chosen tags. Attend ARC205, Protect your critical workloads and recover your most critical asset - your data at Google Cloud Next. View the full article