Search the Community

Most businesses know that taking responsibility for their environmental and social impact is key for long-term success. But how can they make fully-informed decisions when most companies only have visibility into their immediate suppliers? At Prewave, we’re driven by the mission to help companies make their entire supply chains more resilient, transparent, and sustainable. Our end-to-end platform monitors and predicts a wide range of supply chain risks, and AI is the driving-force behind its success. Without AI, handling vast volumes of data and extracting meaningful insights from publicly-available information would be almost unfathomable at the scale that we do to help our clients. Because of that, Prewave needs a rock-solid technology foundation that is reliable, secure, and highly scalable to continually handle this demand. That’s why we built the Prewave supply chain risk intelligence platform on Google Cloud from inception in 2019. Back then, as a small team, we didn’t want to have to maintain hardware or infrastructure, and Google Cloud managed services stood out for providing reliability, availability, and security while freeing us up to develop our product and focus on Prewave’s mission. A shared concern for sustainability also influenced our decision, and we’re proud to be working with data centers with such a low carbon footprint. Tracking hundreds of thousands of suppliers Prewave’s end-to-end platform solves two key challenges for customers: First, it makes supply chains more resilient by identifying description risks and developing the appropriate mitigation plans. And second, it makes supply chains more sustainable by detecting and solving ESG risks, such as forced labor or environmental issues. It all starts with our Risk Monitoring capability, which uses AI that was developed by our co-founder Lisa in 2012 during her PhD research. With it, we’re scanning publicly available information in 120+ languages, looking for insights that can indicate early signals of Risk Events for our clients, such as labor unrest, an accident, fire, or 140 other different risk types that can disrupt their supply chain. Based on the resulting insights, clients can take actions on our platform to mitigate the risk, from filing an incident review to arranging an on-site audit. With this information, Prewave also maps our clients’ supply chains from immediate and sub-tier suppliers down to the raw materials’ providers. Having this level of granularity and transparency is now a requirement of new regulations such as the European corporate sustainability due diligence directive (CSDDD), but it can be challenging for our clients to do without help. They usually have hundreds or thousands of suppliers and our platform helps them to know each one, but also to focus attention, when needed, on those with the highest risk. The Prewave platform keeps effort on the supplier’s side as light as possible. They only have to act if potential risk is flagged by our Tier-N Monitoring capability, in which case, we support them to fix issues and raise their standards. Additionally, this level of visibility frees them up from having to manually answer hundreds of questionnaires in order to qualify to do business with more partners. To make all this possible, our engineering teams rely heavily on scalable technology such as Google Kubernetes Engine (GKE) to support our SaaS. We recently switched from Standard to Autopilot and noticed great results in time efficiency now that we don’t need to ensure that node pools are in place or that all CPU power available is being used appropriately, helping save up to 30% of resources. This also has helped us to reduce costs because we only pay for the deployments we run. We also believe that having the best tools in place is key to delivering the best experience not only to customers but also to our internal teams. So we use Cloud Build and Artifact Registry to experiment, build, and deploy artifacts and manage docker containers that we also use for GKE. Meanwhile, Cloud Armor acts as a firewall protecting us against denial of service and web attacks. Because scalability is key for our purposes, the application development and data science teams use Cloud SQL as a database. This is a fully managed service that helps us focus on developing our product, since we don’t have to worry about managing the servers according to demand. Data science teams also use Compute Engine to host our AI implementations as we develop and maintain our own models, and these systems are at the core of Prewave’s daily work. Helping more businesses improve their deep supply chains Since 2020, Prewave has grown from three clients to more than 170, our team of 10 grew to more than 160, and the company’s revenue growth multiplied by 100, achieving a significant milestone. We’ve also since then released many new features to our platform that required us to scale the product alongside scaling the company. With Google Cloud, this wasn’t an issue. We simply extended the resources that the new implementations needed, helping us to gain more visibility at the right time and win new customers. Because our foundation is highly stable and scalable, growing our business has been a smooth ride. Next, Prewave is continuing its expansion plans into Europe that began in 2023, before moving to new markets, such as the US. This is going well and our association with Google Cloud is helping us win the trust of early-stage clients who clearly also trust in its reliability and security. We’re confident that our collaboration with Google Cloud will continue to bring us huge benefits as we help more companies internationally to achieve transparency, resilience, sustainability, and legal compliance along their deep supply chains. View the full article

Google Cloud Champion Innovators are a global network of more than 600 non-Google professionals, who are technical experts in Google Cloud products and services. Each Champion specializes in one of nine different technical categories which are cloud AI/ML, data analytics, hybrid multi-cloud, modern architecture, security and networking, serverless app development, storage, Workspace and databases. In this interview series we sit down with Champion Innovators across the world to learn more about their journeys, their technology focus, and what excites them. Today we’re talking to Juan Guillermo Gómez. Currently Technical Lead at Wordbox, Juan Guillermo is a Cloud Architect, Google Developer Expert, Serverless App Development Champion, and community builder who is regularly invited to share his thoughts on software architecture, entrepreneurship, and innovation at events across Latin America. Natalie Tack (Google Cloud Editorial): What technology area are you most fascinated with, and why? Juan Guillermo Gómez: As a kid, I dreamed of developing software that would change the world. I've always been fascinated by programming and the idea of creating products and services that make people's lives easier. Nowadays, my focus is on architecture modernization and using innovative new tech to scale and improve systems. Following a Google Cloud hackathon around 10 years ago, I started to take an interest in serverless architecture and became really enthusiastic about Google Cloud serverless computing services, which enable you to focus on coding without worrying about infrastructure. Nowadays, I’m excited about the potential of AI to help us create better, more robust, and more efficient systems, and I'm really looking forward to seeing where that will take us. NT: As a developer, what’s the best way to go about learning new things? JGG: There’s a wealth of resources out there, whether it’s YouTube, podcasts or developer blogs. I find the Google Cloud developer blog and YouTube channel particularly instructive when it comes to real use cases. But the most important thing in my opinion is to be part of a community. It enables you to share experiences, collaborate on projects and learn from others with expertise in specific industries. Google Developer Groups, or GDGs, for example are a great way to network and keep up with the latest developments. Becoming a Google Cloud Champion Innovator has been really beneficial. It enables me to learn directly from Googlers, collaborate around shared problems, and work directly with other people from my field. I can then share that knowledge with my community in Latin America, both as co-organizer of GDG Cali and via my podcast Snippets Tech. NT: Could you tell us more about your experience as an Innovator? JGG: I joined the Innovators program in September 2022, and it was a natural fit from the start. The Innovator culture is deeply rooted in that of developer communities, which I’ve participated in for over 20 years now. The core philosophy is sharing, collaboration and learning from others' experiences, not only through getting together at talks and conferences, but also open source software and libraries. Basically, creating things that benefit the community at large: the Innovator mindset is fundamentally collaborative. As a result, the program provides a wealth of information, and one of the biggest benefits from my point of view is the amount of real use cases it gives access to. I’ve learnt a lot about embedding and semantic similarity that I’m putting into practice in my work as Technical Lead at Wordbox, a startup that helps people learn languages through TV and music. NT: What are some upcoming trends you’re enthusiastic about? JGG: Generally speaking, I’m very interested to see where we’ll go with generative AI. I started working with Google Cloud APIs such as Translate and Speech-to-Text around three years ago as part of my work with Wordbox, and I’m impressed with the way Google Cloud democratizes machine learning and AI, allowing anyone to work with it without extensive machine learning knowledge. There are so many potential use cases for gen AI. As a developer, you can work faster, solve programming language challenges, write unit tests, and refer to best practices. As an Innovator, I gained early access to Duet AI for Developers (now called Gemini Code Assist), which is a great tool to fill in knowledge gaps, suggest scripts and help out junior architects or those that are new to Google Cloud - basically helping you focus on creating great code. NT: Can you tell us about an exciting new project you’re working on? JGG: The Wordbox language learning app features a collection of short videos with English phrases that we show users based on semantic similarity, where the phrase in the new video is similar to the previous one. To enable that, we use Vertex AI, PaLM 2 and Vector Search, and are keen to explore Gemini models, as they offer advanced capabilities for comparing semantic similarities not only between text, but also between text and video, which would enable us to create stories around specific phrases. For example, if a user is watching a video related to a “Game of Thrones” review series and learning certain expressions from it, we can use Gemini models to find similarities with other videos or texts. This will allow us to produce a narrative around the learned expression, creating a comprehensive learning environment that’s tailored to the user’s proficiency level. The learner can then read or watch the story, answer questions and engage in a more interactive, personalized learning experience. As a side project, I’m also working on an AI-enabled platform that helps musicians and singers create lyrics based on keywords, genre, and context. They input the information, and the platform generates lyrics that hopefully serve as a jumping-off point for a great new piece of music. NT: What advice would you give to budding innovators? JGG: The Innovators program can be summed up in three words: networking, learning, and growth. I’d advise anyone interested in becoming an Innovator to be proactive both in learning and in engaging with their fellow developers. It feels pretty great to be given early access to an API and then a few months later tell your community about it while creating fantastic new features for your clients. Take the next steps on your Google Cloud journey and learn more about the Google Cloud Innovators Program, designed to help developers and practitioners grow their Google Cloud skills and advance their careers. No matter where you are on your cloud journey, the Innovators Program has something for you! View the full article

Today, we're launching the general availability (GA) of Direct VPC egress for Cloud Run. This feature enables your Cloud Run resources to send traffic directly to a VPC network without proxying it through Serverless VPC Access connectors, making it easier to set up, faster, and with lower costs. In fact, Direct VPC egress delivers approximately twice the throughput compared to both VPC connectors and the default Cloud Run internet egress path, offering up to 1 GB per second per instance. Whether you're sending traffic to destinations on the VPC, to other Google Cloud services like Cloud Storage, or to other destinations on the public internet, Direct VPC egress offers higher throughput and lower latency for performance-sensitive apps. What's new since the preview Notable improvements and new features: All regions where Cloud Run is available are now enabled for Direct VPC egress. Each Cloud Run service revision with Direct VPC can now scale beyond 100 instances as controlled by a quota. There is a standard quota increase request process if you need to scale even more. Cloud NAT is supported, and Direct VPC egress traffic is now included in VPC Flow Logs and Firewall Rules Logging. These updates address the top issues reported by our preview customers, especially larger customers with advanced scalability, networking, and security requirements. Customer feedback Many customers have been trying Direct VPC egress in preview since last year and have given us great feedback, including DZ BANK: "With Direct VPC egress for Cloud Run, the platform team can more easily onboard new Cloud Run workloads because we no longer need to maintain Serverless VPC Access connectors and their associated dedicated /28 subnets. In our dynamic environment, where new Cloud Run services are created regularly, this simpler networking architecture saves us 4-6 hours per week of manual toil. We have also deprovisioned 30+ VPC connectors, saving on the additional compute costs for running them." - Tim Harpe, Senior Cloud Engineer, DZ BANK If you enable direct VPC egress and send all your egress traffic to a VPC, you can leverage the same tools and capabilities for all your traffic – from Cloud Run, GKE, or VMs. Next steps Direct VPC egress is ready for your production workloads. Try it today and enjoy better performance and lower cost. For a primer about how Direct VPC egress works, check out our preview blog post and its attached explainer video. View the full article

At Ninja Van, running applications in a stable, scalable environment is business-critical. We are a fast-growing tech-enabled express logistics business with operations across South East Asia. In the past 12 months, we’ve partnered with close to two million businesses to deliver around two million parcels daily to customers, with the help of more than 40,000 workers and employees. We are an established customer of Google Cloud, and continue to work closely with them to expand into new products and markets, including supply chain management and last mile courier services. Deploying a secure, stable, container platform To run the applications and microservices architecture that enable our core businesses, we opted to deploy a secure and stable container platform. We had been early adopters of containerization. We were initially running our container workload in CoreOS with a custom scheduler, Fleet. As we monitor the activities in the open source community, it was evident Kubernetes was gaining more traction and is becoming more and more popular. We decided to run our own Kubernetes cluster and API server, later deploying a number of Compute Engine virtual machines, consisting of both control plane and worker nodes. As we dived deeper into Kubernetes, we realized that a lot of what we did was already baked into its core functionalities, such as service discovery. This feature enables applications and microservices to communicate with each other without the need to know where the container is deployed to among the worker nodes. We felt that if we continued maintaining our discovery system, we would just be reinventing the wheel. Thus, we dropped what we had in favor of this Kubernetes core feature. We also found the opportunity to engage with and contribute to the open source community working on Kubernetes compelling. With Kubernetes being open standards-based, we gained the freedom and flexibility to adapt our technology stack to our needs. However, we found upgrading a self-managed Kubernetes challenging and troublesome in the early days and decided to move to a fully managed Kubernetes service. Among other benefits, we could upgrade Kubernetes easily through Google Kubernetes Engine (GKE) user interface or Google Cloud command line tool. We could also enable auto upgrades simply by specifying an appropriate release channel. Simplifying the operation of multiple GKE clusters With a technology stack based on open-source technologies, we have simplified the operation of multiple clusters in GKE. For monitoring and logging operations, we have moved to a centralized architecture to colocate the technology stack on a single management GKE cluster. We use Elasticsearch, Fluentbit and Kibana for logging and Thanos, Prometheus and Grafana for monitoring. When we first started, logging and monitoring were distributed across individual clusters, which meant that administrators had to access the clusters separately which led to a lot of operational inefficiencies. This also meant maintaining duplicated charts across different instances of Kibana and Grafana. For CI/CD, we run a dedicated DevOps GKE cluster used for hosting developer toolsets and running the pipelines. We embrace the Atlassian suite of services, such as JIRA, Confluence, Bitbucket, Bamboo to name a few. These applications are hosted in the same DevOps GKE cluster. Our CI/CD is centralized, but custom steps can be put in, giving some autonomy to teams. When a team pushes out new versions of codes, our CI pipeline undertakes the test and build processes. This then produces container image artifacts for backend services and minified artifacts for frontend websites. Typically, the pipelines are parameterized to push out and test the codes in development and testing environments, and subsequently deploy them into sandbox and production. Depending on their requirements, application teams have the flexibility to opt for a fully automated pipeline or a semi-automated one, that requires manual approval for deployment to production. Autoscaling also comes into play during runtime. As more requests come in, we need to scale up to more containers, but shrink down automatically as the number of requests decreases. To support autoscaling based on our metrics, we integrate KEDA to our technology stack. Delivering a seamless development experience In a sense, our developers are our customers as well as our colleagues. We aim to provide a frictionless experience for them by automating the testing, deployment, management and operation of application services and infrastructure. By doing this, we allow them to focus on building the application they’re assigned. To do this, we’re using DevOps pipelines to automate and simplify infrastructure provisioning and software testing and release cycles. Teams can also self-serve and spin up GKE clusters with the latest environment builds mirroring production with non-production sample data preloaded, which gives them a realistic environment to test the latest fixes and releases. As a build proceeds to deployment, they can visit a Bamboo CI/CD Console to track progress. Code quality is critical to our development process. An engineering excellence sub-vertical within our business monitors code quality through part of our CI/CD using the SonarQube open-source code inspection tool. We set stringent unit test coverage percentage requirements and do not allow anything into our environment that fails unit or integration testing. We release almost once a day excluding code freezes on days like Black Friday or Cyber Monday, when we only release bug fixes or features that need to be deployed urgently during demand peaks. While we’ve not changed our release schedule with GKE, we’ve been able to undertake concurrent builds in parallel environments, enabling us to effectively manage releases across our 200-microservice architecture. Latency key to Google Cloud and GKE selection When we moved from other cloud providers to Google Cloud and GKE, the extremely low latency between Google Cloud data centers, in combination with reliability, scalability and competitive pricing, gave us confidence Google Cloud was the right choice. In a scenario with a lot of people using the website, we can provide a fast response time and a better customer experience. In addition, Google Cloud makes the patching and upgrading of multiple GKE clusters a breeze by automating the upgrade process and proactively informing us when an automated upgrade or change threatens to break one of our APIs. Google Cloud and GKE also open up a range of technical opportunities for our business, including simplification. Many of our services use persistent storage, and GKE provides a simple way to automatically deploy and manage them through their Container Storage Interface (CSI) driver. The CSI driver enables native volume snapshots and in conjunction with Kubernetes CronJob, where we can easily take automated backups of the disks running services such as TiDB, MySQL, Elasticsearch and Kafka. On the development front, we are also exploring Skaffold, which opens up possibilities for development teams to improve their inner development loop cycle and develop more effectively as if their local running instance is deployed within a Kubernetes cluster. Overall, the ease of management of GKE means we can manage our 200-microservice architecture in 15 clusters with just 10 engineers even as the business continues to grow. If you want to try this out yourself, check out a hands-on tutorial where you will set up a continuous delivery pipeline in GKE that automatically rebuilds, retests, and redeploys changes to a sample application. View the full article

We've just added the new service status forums for Amazon Web Services (AWS) & Google Cloud Platform (GCP) https://devopsforum.uk/forum/32-aws-service-status/ https://devopsforum.uk/forum/33-gcp-service-status/ We've also added a new 'block' on the right-hand side of the main homepage, showing the latest 3 statuses 'Cloud Service Status' We've also added an Azure service status forum, however so far there are no posts - apparently If everything is running smoothly on Azure the feed will be empty https://azure.status.microsoft/en-gb/status/feed/ Here are some other status feeds for Azure services; https://azure.status.microsoft/en-us/status/history/ https://feeder.co/discover/d3ca207d93/azure-microsoft-com-en-us-status

Google has always pioneered the development of large and scalable infrastructure to support its search engine and other products. Its vast network servers have enabled us to store and manage immense volumes of data. As cloud computing gained notoriety, Google expanded its operations and launched Google Cloud Platform (GCP). The Google Cloud Storage (GCS) allows […]View the full article

Azure Container Apps Azure container apps is a fully managed Kubernetes service that could be compared to ECS in AWS or Cloud Run in GCP. Compared to AKS, all integrations with Azure are already done for you. The best example is the use of managed identity where here you only need to enable a parameter whereas in AKS it’s complicated and changes every two years. View the full article

Embeddings represent real-world objects, like entities, text, images, or videos as an array of numbers (a.k.a vectors) that machine learning models can easily process. Embeddings are the building blocks of many ML applications such as semantic search, recommendations, clustering, outlier detection, named entity extraction, and more. Last year, we introduced support for text embeddings in BigQuery, allowing machine learning models to understand real-world data domains more effectively and earlier this year we introduced vector search, which lets you index and work with billions of embeddings and build generative AI applications on BigQuery. At Next ’24, we announced further enhancement of embedding generation capabilities in BigQuery with support for: Multimodal embeddings generation in BigQuery via Vertex AI’s multimodalembedding model, which lets you embed text and image data in the same semantic space Embedding generation for structured data using PCA, Autoencoder or Matrix Factorization models that you train on your data in BigQuery Multimodal embeddings Multimodal embedding generates embedding vectors for text and image data in the same semantic space (vectors of items similar in meaning are closer together) and the generated embeddings have the same dimensionality (text and image embeddings are the same size). This enables a rich array of use cases such as embedding and indexing your images and then searching for them via text. You can start using multimodal embedding in BigQuery using the following simple flow. If you like, you can take a look at our overview video which walks through a similar example. Step 0: Create an object table which points to your unstructured data You can work with unstructured data in BigQuery via object tables. For example, if you have your images stored in a Google Cloud Storage bucket on which you want to generate embeddings, you can create a BigQuery object table that points to this data without needing to move it. To follow along the steps in this blog you will need to reuse an existing BigQuery CONNECTION or create a new one following instruction here. Ensure that the principal of the connection used has the ‘Vertex AI User’ role and that the Vertex AI API is enabled for your project. Once the connection is created you can create an object table as follows: code_block <ListValue: [StructValue([('code', "CREATE OR REPLACE EXTERNAL TABLE\r\n `bqml_tutorial.met_images`\r\nWITH CONNECTION `Location.ConnectionID`\r\nOPTIONS\r\n( object_metadata = 'SIMPLE',\r\n uris = ['gs://gcs-public-data--met/*']\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e9805ba60a0>)])]> In this example, we are creating an object table that contains public domain art images from The Metropolitan Museum of Art (a.k.a. “The Met”) using a public Cloud Storage bucket that contains this data. The resulting object table has the following schema: Let’s look at a sample of these images. You can do this using a BigQuery Studio Colab notebook by following instructions in this tutorial. As you can see, the images represent a wide range of objects and art pieces. Image source: The Metropolitan Museum of Art Now that we have the object table with images, let’s create embeddings for them. Step 1: Create model To generate embeddings, first create a BigQuery model that uses the Vertex AI hosted ‘multimodalembedding@001’ endpoint. code_block <ListValue: [StructValue([('code', "CREATE OR REPLACE MODEL\r\n bqml_tutorial.multimodal_embedding_model REMOTE\r\nWITH CONNECTION `LOCATION.CONNNECTION_ID`\r\nOPTIONS (endpoint = 'multimodalembedding@001')"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e9805ba6970>)])]> Note that while the multimodalembedding model supports embedding generation for text, it is specifically designed for cross-modal semantic search scenarios, for example, searching images given text. For text-only use cases, we recommend using the textembedding-gecko@ model instead. Step 2: Generate embeddings You can generate multimodal embeddings in BigQuery via the ML.GENERATE_EMBEDDING function. This function also works for generating text embeddings (via textembedding-gecko model) and structured data embeddings (via PCA, AutoEncoder and Matrix Factorization models). To generate embeddings, simply pass in the embedding model and the object table you created in previous steps to the ML.GENERATE_EMBEDDING function. code_block <ListValue: [StructValue([('code', "CREATE OR REPLACE TABLE `bqml_tutorial.met_image_embeddings`\r\nAS\r\nSELECT * FROM ML.GENERATE_EMBEDDING(\r\n MODEL `bqml_tutorial.multimodal_embedding_model`,\r\n TABLE `bqml_tutorial.met_images`)\r\nWHERE content_type = 'image/jpeg'\r\nLimit 10000"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e9805ba6610>)])]> To reduce the tutorial’s runtime, we limit embedding generation to 10,000 images. This query will take 30 minutes to 2 hours to run. Once this step is completed you can see a preview of the output in BigQuery Studio. The generated embeddings have a dimension of 1408. Step 3 (optional): Create a vector index on generated embeddings While the embeddings generated in the previous step can be persisted and used directly in downstream models and applications, we recommend creating a vector index for improving embedding search performance and enabling the nearest-neighbor query pattern. You can learn more about vector search in BigQuery here. code_block <ListValue: [StructValue([('code', "-- Create a vector index on the embeddings\r\n\r\nCREATE OR REPLACE VECTOR INDEX `met_images_index`\r\nON bqml_tutorial.met_image_embeddings(ml_generate_embedding_result)\r\nOPTIONS(index_type = 'IVF',\r\n distance_type = 'COSINE')"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e9805ba62e0>)])]> Step 4: Use embeddings for text-to-image (cross-modality) search You can now use these embeddings in your applications. For example, to search for “pictures of white or cream colored dress from victorian era” you first embed the search string like so: code_block <ListValue: [StructValue([('code', '-- embed search string\r\n\r\nCREATE OR REPLACE TABLE `bqml_tutorial.search_embedding`\r\nAS\r\nSELECT * FROM ML.GENERATE_EMBEDDING(\r\n MODEL `bqml_tutorial.multimodal_embedding_model`,\r\n (\r\n SELECT "pictures of white or cream colored dress from victorian era" AS content\r\n )\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e9805ba6850>)])]> You can now use the embedded search string to find similar (nearest) image embeddings as follows: code_block <ListValue: [StructValue([('code', '-- use the embedded search string to search for images\r\n\r\nCREATE OR REPLACE TABLE\r\n `bqml_tutorial.vector_search_results` AS\r\nSELECT\r\n base.uri AS gcs_uri,\r\n distance\r\nFROM\r\n VECTOR_SEARCH( TABLE `bqml_tutorial.met_image_embeddings`,\r\n "ml_generate_embedding_result",\r\n TABLE `bqml_tutorial.search_embedding`,\r\n "ml_generate_embedding_result",\r\n top_k => 5)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e980766d250>)])]> Step 5: Visualize results Now let’s visualize the results along with the computed distance and see how we performed on the search query “pictures of white or cream colored dress from victorian era”. Refer the accompanying tutorial on how to render this output using a BigQuery notebook. Image source: The Metropolitan Museum of Art The results look quite good! Wrapping up In this blog, we demonstrated a common vector search usage pattern but there are many other use cases for embeddings. For example, with multimodal embeddings you can perform zero-shot classification of images by converting a table of images and a separate table containing sentence-like labels to embeddings. You can then classify images by computing distance between images and each descriptive label’s embedding. You can also use these embeddings as input for training other ML models, such as clustering models in BigQuery to help you discover hidden groupings in your data. Embeddings are also useful wherever you have free text input as a feature, for example, embeddings of user reviews or call transcripts can be used in a churn prediction model, embeddings of images of a house can be used as input features in a price prediction model etc. You can even use embeddings instead of categorical text data when such categories have semantic meaning, for example, product categories in a deep-learning recommendation model. In addition to multimodal and text embeddings, BigQuery also supports generating embeddings on structured data using PCA, AUTOENCODER and Matrix Factorization models that have been trained on your data in BigQuery. These embeddings have a wide range of use cases. For example, embeddings from PCA and AUTOENCODER models can be used for anomaly detection (embeddings further away from other embeddings are deemed anomalies) and as input features to other models, for example, a sentiment classification model trained on embeddings from an autoencoder. Matrix Factorization models are classically used for recommendation problems, and you can use them to generate user and item embeddings. Then, given a user embedding you can find the nearest item embeddings and recommend these items, or cluster users so that they can be targeted with specific promotions. To generate such embeddings, first use the CREATE MODEL function to create a PCA, AutoEncoder or Matrix Factorization model and pass in your data as input, and then use ML.GENERATE_EMBEDDING function providing the model, and a table input to generate embeddings on this data. Getting started Support for multimodal embeddings and support for embeddings on structured data in BigQuery is now available in preview. Get started by following our documentation and tutorials. Have feedback? Let us know what you think at bqml-feedback@google.com. View the full article

Cloud SQL for SQL Server is a fully managed relational database that offers native SQL Server features along with several Google Cloud innovations, and allows migrating databases while retaining their existing versions or upgrading to newer ones. But migrating SQL Server databases to the cloud can be daunting. Concerns about downtime, complexity, and security risks can hinder your move to managed cloud databases. This week at Google Cloud Next, we announced the preview of support for SQL Server migrations to Cloud SQL for SQL Server in Database Migration Service , a fully managed serverless cloud service that performs database migrations with minimal downtime. Database Migration Service reduces migration complexity by loading an initial snapshot of the database followed by non-intrusive incremental load, reducing operational downtime. Let’s take a look at how it works and how to get started. Introducing SQL Server migration support Database Migration Service leverages built-in backup and restore to facilitate high-fidelity, low-downtime SQL Server migrations. Similar to other Database Migration Service homogeneous migration paths, SQL Server to Cloud SQL for SQL Server migrations are secure and reliable, and available at no additional charge. Database Migration Service offers: Minimal downtime and system overhead : Database Migration Service’s continuous native backup and restore technology helps ensure that your SQL Server source databases remain operational while the migration progresses, for reduced downtime. Because Database Migration Service leverages SQL Server backups to perform the migration, no additional overhead is introduced to your production operational systems. Serverless simplicity: As a managed service, you don’t need to manage any infrastructure for Database Migration Service, allowing your IT teams to focus on strategic initiatives rather than on managing complex migration setups. Security at the forefront: Database Migration Service supports encrypted SQL Server backups for robust protection throughout the migration. At Google Cloud, we’ve been working closely with customers who are looking to migrate SQL Server workloads. One of them is GoodRx Holdings, Inc., an American healthcare company that operates a telemedicine platform. "GoodRx is excited that Database Migration Service is now available to migrate SQL Server to Cloud SQL for SQL Server. Google Cloud's solutions help our business improve operational efficiencies in an intelligent, innovative, and cost-effective way." - Nitin Shingate, CTO GoodRx Migrating your SQL Server Databases with Database Migration Service Here’s how to start migrating your SQL Server workloads today using Database Migration Service: Navigate to the Database Migration area of the Google Cloud console, under Databases, and create a Connection Profile pointing to a Cloud Storage bucket to which your SQL Server backup files will be uploaded. Similarly, create another Connection Profile pointing to your chosen Cloud SQL for SQL Server instance. Create a migration job to connect both connection profiles and choose which databases you’d like to migrate. Test your migration job and make sure the test was successful as displayed below.,Then start it whenever you're ready. 5. Upload a full backup and continuously upload transaction log backups for each database to the Cloud Storage bucket at your desired frequency. 6. Once the full backups have been restored, the migration phase will change from “Initial Load” to “Incremental Load”. Database Migration Service keeps looking for new transaction log backup files and continuously restore them. 7. Monitor the progress of your migration jobs using the built-in metrics and logs to assess the right moment to complete the migration. 8. When ready to complete the migration, click “Promote” for Database Migration Service to recover the databases, making them available and completing the migration. Congratulations! You’ve completed your migration to Cloud SQL for SQL Server. Get started today Ready to migrate? Start your SQL Server migration to Google Cloud with the Database Migration Service. View the full article

Delta Lake is an open-source optimized storage layer that provides a foundation for tables in lake houses and brings reliability and performance improvements to existing data lakes. It sits on top of your data lake storage (like cloud object stores) and provides a performant and scalable metadata layer on top of data stored in the Parquet format. Organizations use BigQuery to manage and analyze all data types, structured and unstructured, with fine-grained access controls. In the past year, customer use of BigQuery to process multiformat, multicloud, and multimodal data using BigLake has grown over 60x. Support for open table formats gives you the flexibility to use existing open source and legacy tools while getting the benefits of an integrated data platform. This is enabled via BigLake — a storage engine that allows you to store data in open file formats on cloud object stores such as Google Cloud Storage, and run Google-Cloud-native and open-source query engines on it in a secure, governed, and performant manner. BigLake unifies data warehouses and lakes by providing an advanced, uniform data governance model. This week at Google Cloud Next '24, we announced that this support now extends to the Delta Lake format, enabling you to query Delta Lake tables stored in Cloud Storage or Amazon Web Services S3 directly from BigQuery, without having to export, copy, nor use manifest files to query the data. Why is this important? If you have existing dependencies on Delta Lake and prefer to continue utilizing Delta Lake, you can now leverage BigQuery native support. Google Cloud provides an integrated and price-performant experience for Delta Lake workloads, encompassing unified data management, centralized security, and robust governance. Many customers already harness the capabilities of Dataproc or Serverless Spark to manage Delta Lake tables on Cloud Storage. Now, BigQuery’s native Delta Lake support enables seamless delivery of data for downstream applications such as business intelligence, reporting, as well as integration with Vertex AI. This lets you do a number of things, including: Build a secure and governed lakehouse with BigLake’s fine-grained security model Securely exchange Delta Lake data using Analytics Hub Run data science workloads on Delta Lake using BigQuery ML and Vertex AI How to use Delta Lake with BigQuery Delta Lake tables follow the same table creation process as BigLake tables. Required roles To create a BigLake table, you need the following BigQuery identity and access management (IAM) permissions: bigquery.tables.create bigquery.connections.delegate Prerequisites Before you create a BigLake table, you need to have a dataset and a Cloud resource connection that can access Cloud Storage. Table creation using DDL Here is the DDL statement to create a Delta lake Table code_block <ListValue: [StructValue([('code', 'CREATE EXTERNAL TABLE `PROJECT_ID.DATASET.DELTALAKE_TABLE_NAME`\r\nWITH CONNECTION `PROJECT_ID.REGION.CONNECTION_ID`\r\nOPTIONS (\r\n format ="DELTA_LAKE",\r\n uris=[\'DELTA_TABLE_GCS_BASE_PATH\']);'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e9803cb44f0>)])]> Querying Delta Lake tables After creating a Delta Lake BigLake table, you can query it using GoogleSQL syntax, the same as you would a standard BigQuery table. For example: code_block <ListValue: [StructValue([('code', 'SELECT FIELD1, FIELD2 FROM `PROJECT_ID.DATASET.DELTALAKE_TABLE_NAME`'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3e9803cb4550>)])]> You can also enforce fine-grained security at the table level, including row-level and column-level security. For Delta Lake tables based on Cloud Storage, you can also use dynamic data masking. Conclusion We believe that BigQuery’s support for Delta Lake is a major step forward for customers building lakehouses using Delta Lake. This integration will make it easier for you to get insights from your data and make data-driven decisions. We are excited to see how you use Delta Lake and BigQuery together to solve their business challenges. For more information on how to use Delta Lake with BigQuery, please refer to the documentation. Acknowledgments: Mahesh Bogadi, Garrett Casto, Yuri Volobuev, Justin Levandoski, Gaurav Saxena, Manoj Gunti, Sami Akbay, Nic Smith and the rest of the BigQuery Engineering team. View the full article

Today, we are announcing the general availability of provider-defined functions in the AWS, Google Cloud, and Kubernetes providers in conjunction with the HashiCorp Terraform 1.8 launch. This release represents yet another step forward in our unique approach to ecosystem extensibility. Provider-defined functions will allow anyone in the Terraform community to build custom functions within providers and extend the capabilities of Terraform. Introducing provider-defined functions Previously, users relied on a handful of built-in functions in the Terraform configuration language to perform a variety of tasks, including numeric calculations, string manipulations, collection transformations, validations, and other operations. However, the Terraform community needed more capabilities than the built-in functions could offer. With the release of Terraform 1.8, providers can implement custom functions that you can call from the Terraform configuration. The schema for a function is defined within the provider's schema using the Terraform provider plugin framework. To use a function, declare the provider as a required_provider in the terraform{} block: terraform { required_version = ">= 1.8.0" required_providers { time = { source = "hashicorp/local" version = "2.5.1" } } }Provider-defined functions can perform multiple tasks, including: Transforming existing data Parsing combined data into individual, referenceable components Building combined data from individual components Simplifying validations and assertions To access a provider-defined function, reference the provider:: namespace with the local name of the Terraform Provider. For example, you can use the direxists function by including provider::local::direxists() in your Terraform configuration. Below you’ll find several examples of new provider-defined functions in the officially supported AWS, Google Cloud, and Kubernetes providers. Terraform AWS provider The 5.40 release of the Terraform AWS provider includes its first provider-defined functions to parse and build Amazon Resource Names (ARNs), simplifying Terraform configurations where ARN manipulation is required. The arn_parse provider-defined function is used to parse an ARN and return an object of individual referenceable components, such as a region or account identifier. For example, to get the AWS account ID from an Amazon Elastic Container Registry (ECR) repository, use the arn_parse function to retrieve the account ID and set it as an output: # create an ECR repository resource "aws_ecr_repository" "hashicups" { name = "hashicups" image_scanning_configuration { scan_on_push = true } } # output the account ID of the ECR repository output "hashicups_ecr_repository_account_id" { value = provider::aws::arn_parse(aws_ecr_repository.hashicups.arn).account_id } Running terraform apply against the above configuration outputs the AWS Account ID: Apply complete! Resources: 2 added, 0 changed, 0 destroyed. Outputs: hashicups_ecr_repository_account_id = "751192555662" Without the arn_parse function, you would need to define and test a combination of built-in Terraform functions to split the ARN and reference the proper index or define a regular expression to match on a substring. The function handles the parsing for you in a concise manner so that you do not have to worry about doing this yourself. The AWS provider also includes a new arn_build function that builds an ARN from individual attributes and returns it as a string. This provider-defined function can create an ARN that you cannot reference from another resource. For example, you may want to allow another account to pull images from your ECR repository. The arn_build function below constructs an ARN for an IAM policy using an account ID: # allow another account to pull from the ECR repository data "aws_iam_policy_document" "cross_account_pull_ecr" { statement { sid = "AllowCrossAccountPull" effect = "Allow" principals { type = "AWS" identifiers = [ provider::aws::arn_build("aws", "iam", "", var.cross_account_id, "root"), ] } actions = [ "ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer", ] } }The arn_build function helps to guide and simplify the process of combining substrings to form an ARN, and it improves readability compared to using string interpolation. Without it, you'd have to look up the exact ARN structure in the AWS documentation and manually test it. Terraform Google Cloud provider The 5.23 release of the Terraform Google Cloud provider adds a simplified way to get regions, zones, names, and projects from the IDs of resources that aren’t managed by your Terraform configuration. Provider-defined functions can now help parse Google IDs when adding an IAM binding to a resource that’s managed outside of Terraform: resource "google_cloud_run_service_iam_member" "example_run_invoker_jane" { member = "user:jane@example.com" role = "run.invoker" service = provider::google::name_from_id(var.example_cloud_run_service_id) location = provider::google::location_from_id(var.example_cloud_run_service_id) project = provider::google::project_from_id(var.example_cloud_run_service_id) }The Google Cloud provider also includes a new region_from_zone provider-defined function that helps obtain region names from a given zone (e.g. “us-west1” from “us-west1-a”). This simple string processing could be achieved in multiple ways using Terraform’s built-in functions previously, but the new function simplifies the process: locals { zone = “us-central1-a” # ways to derive the region “us-central1” using built-in functions region_1 = join("-", slice(split("-", local.zone), 0, 2)) region_2 = substr(local.zone, 0, length(local.zone)-2) # our new region_from_zone function makes this easier! region_3 = provider::google::region_from_zone(local.zone) }Terraform Kubernetes provider The 2.28 release of the Terraform Kubernetes provider includes provider-defined functions for encoding and decoding Kubernetes manifests into Terraform, making it easier for practitioners to work with the kubernetes_manifest resource. Users that have a Kubernetes manifest in YAML format can use the manifest_decode function to convert it into a Terraform object. The example below shows how to use the manifest_decode function by referring to a Kubernetes manifest in YAML format embedded in the Terraform configuration: locals { manifest = <If you prefer to decode a YAML file instead of using an embedded YAML format, you can do so by combining the built-in file function with the manifest_decode function. $ cat manifest.yaml --- kind: Namespace apiVersion: v1 metadata: name: test labels: name: testresource "kubernetes_manifest" "example" { manifest = provider::kubernetes::manifest_decode(file("${path.module}/manifest.yaml")) }If your manifest YAML contains multiple Kubernetes resources, you may use the manifestdecodemulti function to decode them into a list which can then be used with the for_each attribute on the kubernetes_manifest resource: $ cat manifest.yaml --- kind: Namespace apiVersion: v1 metadata: name: test-1 labels: name: test-1 --- kind: Namespace apiVersion: v1 metadata: name: test-2 labels: name: test-2 resource "kubernetes_manifest" "example" { for_each = { for m in provider::kubernetes::manifest_decode_multi(file("${path.module}/manifest.yaml"))): m.metadata.name => m } manifest = each.value }Getting started with provider-defined functions Provider-defined functions allow Terraform configurations to become more expressive and readable by declaring practitioner intent and reducing complex, repetitive expressions. To learn about all of the new launch-day provider-defined functions, please review the documentation and changelogs of the aforementioned providers: Terraform AWS provider Terraform Google provider Terraform Kubernetes provider Review our Terraform Plugin Framework documentation to learn more about how provider-defined functions work and how you can make your own. We are thankful to our partners and community members for their valuable contributions to the HashiCorp Terraform ecosystem. View the full article

Some Google Cloud customers will be able to run instances on the Arm-based Axion chip later this year. View the full article

Welcome to our live coverage of Google Cloud Next 2024! We're live in Las Vegas for a packed week of news and announcements from one of the world's biggest players when it comes to cloud innovation. The conference starts this morning with an opening keynote from Google Cloud CEO Thomas Kurian, and we've been promised plenty of other special guests. We'll be keeping you updated with all the latest announcements here, so stay tuned for everything you need to know for Google Cloud Next 2024! So what are we expecting to see from Google Cloud this week? In a word - AI. The company has been bullish in its use of AI across the board, and has been ramping up Gemini and Vertex in many of its product lines. Elsewhere, we're probably set to see a host of new updates and upgrades for Google Workspace - again, most likely to be around AI, but with the scale of the office software suite, who knows? We may also see some new hardware - with Nvidia recently announcing its latest flagship Blackwell chips, it's very likely we'll see something with Google announced here. Good morning from a beautifully sunny Las Vegas (although our view of the US eclipse yesterday was...sub-par, it has to be said) We're getting ready to head over for the day one keynote, hosted by none other than Google Cloud CEO Thomas Kurian himself. With a star-studded lineup of customers, partners and guests, the keynote should also include a whole host of news and announcements, so stay tuned for more soon! View the full article

We’re entering a new era for data analytics, going from narrow insights to enterprise-wide transformation through a virtuous cycle of data, analytics, and AI. At the same time, analytics and AI are becoming widely accessible, providing insights and recommendations to anyone with a question. Ultimately, we’re going beyond our own human limitations to leverage AI-based data agents to find deeply hidden insights for us. Organizations already recognize that data and AI can come together to unlock the value of AI for their business. Research from Google’s 2024 Data and AI Trends Report highlighted 84% of data leaders believe that generative AI will help their organization reduce time-to-insight, and 80% agree that the lines of data and AI are starting to blur. Today at Google Cloud Next ’24, we’re announcing new innovations for BigQuery and Looker that will help activate all of your data with AI: BigQuery is a unified AI-ready data platform with support for multimodal data, multiple serverless processing engines and built-in streaming and data governance to support the entire data-to-AI lifecycle. New BigQuery integrations with Gemini models in Vertex AI support multimodal analytics, vector embeddings, and fine-tuning of LLMs from within BigQuery, applied to your enterprise data. Gemini in BigQuery provides AI-powered experiences for data preparation, analysis and engineering, as well as intelligent recommenders to optimize your data workloads. Gemini in Looker enables business users to chat with their enterprise data and generate visualizations and reports—all powered by the Looker semantic data model that’s seamlessly integrated into Google Workspace. Let’s take a deeper look at each of these developments. BigQuery: the unified AI-ready data foundation BigQuery is now Google Cloud’s single integrated platform for data to AI workloads. BigLake, BigQuery’s unified storage engine, provides a single interface across BigQuery native and open formats for analytics and AI workloads, giving you the choice of where your data is stored and access to all of your data, whether structured or unstructured, along with a universal view of data supported by a single runtime metastore, built-in governance, and fine grained access controls. Today we’re expanding open format support with the preview of a fully managed experience for Iceberg, with DDL, DML and high throughput support. In addition to support for Iceberg and Hudi, we’re also extending BigLake capabilities with native support for the Delta file format, now in preview. “At HCA Healthcare we are committed to the care and improvement of human life. We are on a mission to redesign the way care is delivered, letting clinicians focus on patient care and using data and AI where it can best support doctors and nurses. We are building our unified data and AI foundation using Google Cloud's lakehouse stack, where BigQuery and BigLake enable us to securely discover and manage all data types and formats in a single platform to build the best possible experiences for our patients, doctors, and nurses. With our data in Google Cloud’s lakehouse stack, we’ve built a multimodal data foundation that will enable our data scientists, engineers, and analysts to rapidly innovate with AI." - Mangesh Patil, Chief Analytics Officer, HCA Healthcare We’re also extending our cross-cloud capabilities of BigQuery Omni. Through partnerships with leading organizations like Salesforce and our recent launch of bidirectional data sharing between BigQuery and Salesforce Data Cloud, customers can securely combine data across platforms with zero copy and zero ops to build AI models and predictions on combined Salesforce and BigQuery data. Customers can also enrich customer 360 profiles in Salesforce Data Cloud with data from BigQuery, driving additional personalization opportunities powered by data and AI. “It is great to collaborate without boundaries to unlock trapped data and deliver amazing customer experiences. This integration will help our joint customers tap into Salesforce Data Cloud's rich capabilities and use zero copy data sharing and Google AI connected to trusted enterprise data.” - Rahul Auradkar, EVP and General Manager of United Data Services & Einstein at Salesforce Building on this unified AI-ready data foundation, we are now making BigQuery Studio generally available, which already has hundreds of thousands of active users. BigQuery Studio provides a collaborative data workspace across data and AI that all data teams and practitioners can use to accelerate their data-to-AI workflows. BigQuery Studio provides the choice of SQL, Python, Spark or natural language directly within BigQuery, as well as new integrations for real-time streaming and governance. Customers’ use of serverless Apache Spark for data processing increased by over 500% in the past year1. Today, we are excited to announce the preview of our serverless engine for Apache Spark integrated within BigQuery Studio to help data teams work with Python as easily as they do with SQL, without having to manage infrastructure. The data team at Snap Inc. uses these new capabilities to converge toward a common data and AI platform with multiple engines that work across a single copy of data. This gives them the ability to enforce fine-grained governance and track lineage close to the data to easily expand analytics and AI use cases needed to drive transformation. To make data processing on real-time streams directly accessible from BigQuery, we’re announcing the preview of BigQuery continuous queries providing continuous SQL processing over data streams, enabling real-time pipelines with AI operators or reverse ETL. We are also announcing the preview of Apache Kafka for BigQuery as a managed service to enable streaming data workloads based on open-source APIs. We’re expanding our governance capabilities with Dataplex with new innovations for data-to-AI governance available in preview. You can now perform integrated search and drive gen AI-powered insights on your enterprise data, including data and models from Vertex AI, with a fully integrated catalog in BigQuery. We’re introducing column-level lineage in BigQuery and expanding lineage capabilities to support Vertex AI pipelines (available in preview soon) to help you better understand data-to-AI workloads. Finally, to facilitate governance for data-access at scale, we are launching governance rules in Dataplex. Multimodal analytics with new BigQuery and Vertex AI integrations With BigQuery’s direct integration with Vertex AI, we are now announcing the ability to connect models in Vertex AI with your enterprise data, without having to copy or move your data out of BigQuery. This enables multi-modal analytics using unstructured data, fine tuning of LLMs and the use of vector embeddings in BigQuery. Priceline, for instance, is using business data stored in BigQuery for LLMs across a wide range of applications. “BigQuery gave us a solid data foundation for AI. Our data was exactly where we needed it. We were able to connect millions of customer data points from hotel information, marketing content, and customer service chat and use our business data to ground LLMs.” - Allie Surina Dixon, Director of Data, Priceline The direct integration between BigQuery and Vertex AI now enables seamless preparation and analysis of multimodal data such as documents, audio and video files. BigQuery features rich support for analyzing unstructured data using object tables and Vertex AI Vision, Document AI and Speech-to-Text APIs. We are now enabling BigQuery to analyze images and video using Gemini 1.0 Pro Vision, making it easier than ever to combine structured with unstructured data in data pipelines using the generative AI capabilities of the latest Gemini models. BigQuery makes it easier than ever to execute AI on enterprise data by providing the ability to build prompts based on your BigQuery data, and use of LLMs for sentiment extraction, classification, topic detection, translation, classification, data enrichment and more. BigQuery now also supports generating vector embeddings and indexing them at scale using vector and semantic search. This enables new use cases that require similarity search, recommendations or retrieval of your BigQuery data, including documents, images or videos. Customers can use the semantic search in the BigQuery SQL interface or via our integration with gen AI frameworks such as LangChain and leverage Retrieval Augmented Generation based on their enterprise data. Gemini in BigQuery and Gemini in Looker for AI-powered assistance Gen AI is creating new opportunities for rich data-driven experiences that enable business users to ask questions, build custom visualizations and reports, and surface new insights using natural language. In addition to business users, gen AI assistive and agent capabilities can also accelerate the work of data teams, spanning data exploration, analysis, governance, and optimization. In fact, more than 90% of organizations believe business intelligence and data analytics will change significantly due to AI. Today, we are announcing the public preview of Gemini in BigQuery, which provides AI-powered features that enhance user productivity and optimize costs throughout the analytics lifecycle, from ingestion and pipeline creation to deriving valuable insights. What makes Gemini in BigQuery unique is its contextual awareness of your business through access to metadata, usage data, and semantics. Gemini in BigQuery also goes beyond chat assistance to include new visual experiences such as data canvas, a new natural language-based experience for data exploration, curation, wrangling, analysis, and visualization workflows. Imagine you are a data analyst at a bikeshare company. You can use the new data canvas of Gemini in BigQuery to explore the datasets, identify the top trips and create a customized visualization, all using natural language prompts within the same interface Gemini in BigQuery capabilities extend to query recommendations, semantic search capabilities, low-code visual data pipeline development tools, and AI-powered recommendations for query performance improvement, error minimization, and cost optimization. Additionally, it allows users to create SQL or Python code using natural language prompts and get real-time suggestions while composing queries. Today, we are also announcing the private preview of Gemini in Looker to enable business users and analysts to chat with their business data. Gemini in Looker capabilities include conversational analytics, report and formula generation, LookML and visualization assistance, and automated Google slide generation. What’s more, these capabilities are being integrated with Workspace to enable users to easily access beautiful data visualizations and insights right where they work. Imagine you’re an ecommerce store. You can query Gemini in Looker to learn sales trends and market details and immediately explore the insights, with details on how the charts were created. To learn more about our data analytics product innovations, hear customer stories, and gain hands-on knowledge from our developer experts, join our data analytics spotlights and breakout sessions at Google Cloud Next ‘24, or watch them on-demand. 1. Google internal data - YoY growth of data processed using Apache Spark on Google Cloud compared with Feb ‘23 View the full article

Protecting people, data, and critical assets is a crucial responsibility for modern organizations, yet conventional security approaches often struggle to address the escalating velocity, breadth, and intricacy of modern cyberattacks. Bolting on more new security products is simply not a viable long-term strategy. What organizations need from their security solutions is a convergence of essential capabilities that brings simplicity, streamlines operations, and enhances efficiency and effectiveness. Today at Google Cloud Next, we are announcing innovations across our security portfolio that are designed to deliver stronger security outcomes and enable every organization to make Google a part of their security team. Increasing speed and productivity with Gemini in Security Generative AI offers tremendous potential to tip the balance in favor of defenders and we continue to infuse AI-driven capabilities into our products. Today we’re announcing the following new AI capabilities: Gemini in Security Operations is coming to the entire investigation lifecycle, building on our December GA of natural language search and case summaries in Chronicle. A new assisted investigation feature, generally available at the end of this month, will guide analysts through their workflow wherever they are in Chronicle Enterprise and Chronicle Enterprise Plus. Gemini recommends actions based on the context of an investigation, and can run searches and create detection rules to improve response times. Plus, analysts can now ask Gemini for the latest threat intelligence from Mandiant directly in-line — including any indicators of compromise found in their environment — and Gemini will navigate users to the most relevant pages in the integrated platform for deeper investigation. Gemini in Security Operations allows users to quickly investigate incidents and alerts using conversational chat in Chronicle. “Gemini in Security Operations is enabling us to enhance the efficiency of our Cybersecurity Operations Center program as we continue to drive operational excellence,” said Ronald Smalley, senior vice president of Cybersecurity Operations, Fiserv. “Detection engineers can create detections and playbooks with less effort, and security analysts can find answers quickly with intelligent summarization and natural language search. This is critical as SOC teams continue to manage increasing data volumes and need to detect, validate, and respond to events faster.“ Gemini in Threat Intelligence now offers conversational search across Mandiant’s vast and growing repository of threat intelligence directly from frontline investigations — a grounded experience, now in preview. Plus, VirusTotal now automatically ingests OSINT reports, which Gemini summarizes directly in the platform — a feature that’s generally available now. Gemini in Security Command Center now offers preview features that let security teams search for threats and other security events using natural language. It can also provide summaries of critical- and high-priority misconfiguration and vulnerability alerts, and summarize attack paths to help understand cloud risks for remediation. We are also infusing AI in many of our cloud platform’s security services. Today, we’re announcing previews of new capabilities in Gemini Cloud Assist, including: IAM Recommendations, which can provide straightforward, contextual recommendations to remove roles from over-permissioned users or service accounts to help uplevel IAM posture and reduce risk exposure. Key Insights, which can provide assistance during encryption key creation based on its understanding of your data, your encryption preferences, and your compliance needs. Confidential Computing Insights, which can recommend options for adding confidential computing protection for your most sensitive workloads based on your data and your compute usage. Delivering a new frontline of defense for the enterprise Chrome Enterprise Premium is a new offering that redefines, simplifies, and strengthens endpoint security. It brings together the most popular and trusted enterprise browser with Google’s advanced security capabilities, including threat and data protection, Zero Trust access controls, enterprise policy controls, and security insights and reporting. With Chrome Enterprise Premium, which is generally available today, hundreds of millions of enterprise users can get additional protection delivered instantly where they do their work every day. "With Chrome Enterprise Premium, we have confidence in Google’s security expertise, including Project Zero’s cutting-edge security research and fast security patches. We set up data loss prevention restrictions and warnings for sharing sensitive information in applications like Generative AI platforms and noticed a noteworthy 50% reduction in content transfers,” said Nick Reva, head of corporate security engineering, Snap Inc. Turning intelligence into action Our focus on intelligence-driven outcomes continues with the launch of Applied Threat Intelligence in Google Security Operations. Applied threat intelligence takes our industry-leading global threat visibility and automatically applies it to each customer’s unique environment. It can help security operations teams uncover more threats with less effort and use the most up-to-date threat intelligence to address them before they create damage or loss. Managing cloud risk Security Command Center Enterprise, the industry’s first cloud risk-management solution that fuses proactive cloud security and enterprise security operations, is now generally available. This new solution offers security teams a single view of their posture controls, active threats, cloud identities, data, and more, while integrating remediation and issue accountability into end-to-end workflows. Mandiant Hunt for Security Command Center Enterprise is now in preview, and offers on-demand human expertise that can become an extension of internal security operations teams. Hundreds of elite-level analysts and researchers are available on-call to proactively find elusive threats in organizations’ SCC environments. New security capabilities in our trusted cloud We continue our regular delivery of new security controls and capabilities on our cloud platform to help organizations meet evolving policy, compliance, and business objectives. Today we’re announcing the following updates: For Identity and Access Management: Privileged Access Manager (PAM), now available in preview, is designed to help mitigate risks tied to privileged access misuse or abuse. PAM can help customers shift from always-on standing privileges towards on-demand access with just-in-time, time-bound, and approval-based access elevations. Principal Access Boundary (PAB) is a new, identity-centric control now in preview. It can empower security administrators to enforce restrictions on IAM principals so that they can only access authorized resources within a specific defined boundary. For Network Security: Cloud NGFW Enterprise is now generally available. Our cloud-first next generation firewall (NGFW) includes threat protection powered by Palo Alto Networks with a unique distributed architecture that can provide granular control at the workload level. Cloud Armor Enterprise, now generally available, offers a pay-as-you-go model that includes advanced network DDoS protection, web application firewall capabilities, network edge policy, adaptive protection, and threat intelligence to help protect your cloud applications and services. For Data Security: Confidential Accelerators: Confidential VMs on Intel TDX are now in preview and available on the C3 machine series with Intel TDX. For AI and ML workloads, we support Intel AMX, which provides CPU-based acceleration by default on C3 series Confidential VMs. In addition, Confidential Compute will also be coming to A3 VMs with NVIDIA H100 GPUs in preview later this year. With these announcements, our Confidential Computing portfolio now spans Intel, AMD, and NVIDIA hardware. Sensitive Data Protection integration with Cloud SQL is now generally available, and is deeply integrated into the Security Command Center Enterprise risk engine. This powerful combination can pinpoint high-value assets, analyze vulnerabilities in databases, and simulate real-world attack scenarios that can enable you to proactively address risks and safeguard data. Key management with Autokey is now in preview. Autokey simplifies creating and managing customer encryption keys (CMEK) by ensuring you use the right key type for each resource, thus reducing complexity and management overhead. Plus, Autokey can help you adhere to industry best practices for data security. Expanded regions available for bare metal HSM deployments allows you to deploy your own HSMs in PCI-compliant facilities with your Google Cloud workloads. For our Regulated Cloud offerings: Regional Controls for Assured Workloads is now in preview and is available in 32 cloud regions in 14 countries. The Regional Controls package can enforce data residency for customer content at rest, offers administrative access transparency, as well as compliant service restriction and monitoring. Regional controls are available at no additional cost. Audit Manager is now in preview. Audit Manager can help customers drastically simplify their compliance audit process by automating control verification with proof of compliance for their workloads and data on Google Cloud. Take your next security steps with Google Cloud Google’s decade of AI innovation, coupled with our security expertise, means we are strongly positioned to help you protect your users and brand by becoming an integral part of your security team. For more on our Next ‘24 announcements, you can watch our security spotlight, and check out the many great security breakout sessions at Google Cloud Next — live or on-demand. View the full article

Multi-cluster Ingress (MCI) is an advanced feature typically used in cloud computing environments that enables the management of ingress (the entry point for external traffic into a network) across multiple Kubernetes clusters. This functionality is especially useful for applications that are deployed globally across several regions or clusters, offering a unified method to manage access to these applications. MCI simplifies the process of routing external traffic to the appropriate cluster, enhancing both the reliability and scalability of applications. Here are key features and benefits of Multi-cluster Ingress: Global Load Balancing: MCI can intelligently route traffic to different clusters based on factors like region, latency, and health of the service. This ensures users are directed to the nearest or best-performing cluster, improving the overall user experience. Centralized Management: It allows for the configuration of ingress rules from a single point, even though these rules are applied across multiple clusters. This simplification reduces the complexity of managing global applications. High Availability and Redundancy: By spreading resources across multiple clusters, MCI enhances the availability and fault tolerance of applications. If one cluster goes down, traffic can be automatically rerouted to another healthy cluster. Cross-Region Failover: In the event of a regional outage or a significant drop in performance within one cluster, MCI can perform automatic failover to another cluster in a different region, ensuring continuous availability of services. Cost Efficiency: MCI helps optimize resource utilization across clusters, potentially leading to cost savings. Traffic can be routed to clusters where resources are less expensive or more abundant. Simplified DNS Management: Typically, MCI solutions offer integrated DNS management, automatically updating DNS records based on the health and location of clusters. This removes the need for manual DNS management in a multi-cluster setup. How What is Multi-cluster Ingress (MCI) works? Multi-cluster Ingress (MCI) works by managing and routing external traffic into applications running across multiple Kubernetes clusters. This process involves several components and steps to ensure that traffic is efficiently and securely routed to the appropriate destination based on predefined rules and policies. Here’s a high-level overview of how MCI operates: 1. Deployment Across Multiple Clusters Clusters Preparation: You deploy your application across multiple Kubernetes clusters, often spread across different geographical locations or cloud regions, to ensure high availability and resilience. Ingress Configuration: Each cluster has its own set of resources and services that need to be exposed externally. With MCI, you configure ingress resources that are aware of the multi-cluster environment. 2. Central Management and Configuration Unified Ingress Control: A central control plane is used to manage the ingress resources across all participating clusters. This is where you define the rules for how external traffic should be routed to your services. DNS and Global Load Balancer Setup: MCI integrates with global load balancers and DNS systems to direct users to the closest or most appropriate cluster based on various criteria like location, latency, and the health of the clusters. 3. Traffic Routing Initial Request: When a user makes a request to access the application, the DNS resolution directs the request to the global load balancer. Global Load Balancing: The global load balancer evaluates the request against the configured routing rules and the current state of the clusters (e.g., load, health). It then selects the optimal cluster to handle the request. Cluster Selection: The criteria for cluster selection can include geographic proximity to the user, the health and capacity of the clusters, and other custom rules defined in the MCI configuration. Request Forwarding: Once the optimal cluster is selected, the global load balancer forwards the request to an ingress controller in that cluster. Service Routing: The ingress controller within the chosen cluster then routes the request to the appropriate service based on the path, host, or other headers in the request. 4. Health Checks and Failover Continuous Monitoring: MCI continuously monitors the health and performance of all clusters and their services. This includes performing health checks and monitoring metrics to ensure each service is functioning correctly. Failover and Redundancy: In case a cluster becomes unhealthy or is unable to handle additional traffic, MCI automatically reroutes traffic to another healthy cluster, ensuring uninterrupted access to the application. 5. Scalability and Maintenance Dynamic Scaling: As traffic patterns change or as clusters are added or removed, MCI dynamically adjusts routing rules and load balancing to optimize performance and resource utilization. Configuration Updates: Changes to the application or its deployment across clusters can be managed centrally through the MCI configuration, simplifying updates and maintenance. Example Deployment YAML for Multi-cluster Ingress with FrontendConfig and BackendConfig This example includes: A simple web application deployment. A service to expose the application within the cluster. A MultiClusterService to expose the service across clusters. A MultiClusterIngress to expose the service externally with FrontendConfig and BackendConfig. The post What is Multi-cluster Ingress (MCI) appeared first on DevOpsSchool.com. View the full article

Google Cloud Next ‘24 is right around the corner, and this year, there will be even more for developers to enjoy. Whether you're on the generative AI kick or keeping up with what's new with Javascript frameworks, there's a session for technical practitioners in the Application Developers Zone! Meet the experts and get inspired to solve your biggest technical challenges over three days in Las Vegas, April 9-11. With over 600 sessions available, you can choose from a wide range of topics, covering everything from security to growing revenue with AI. Here are a few of the sessions you won't want to miss: 1. Building generative AI apps on Google Cloud with LangChain Learn how to use LangChain, the most popular open-source framework for building LLM-based apps, on Cloud Run to add gen AI features to your own applications. See how to combine it with Cloud SQL's pgvector extension for vector storage to quickly locate similar data points with vector search and reduce the cost of deploying models with Vertex Endpoints. 2. How to deploy all the JavaScript frameworks to Cloud Run Join this session, where we’ll deploy as many JavaScript frameworks as we can, including Angular, React, and Node.js, as quickly as we can to prove that it can be done and have some fun! 3. Build full stack applications using Firebase & Google Cloud Discover how Firebase and Google Cloud simplify full-stack development, empowering you to rapidly build scalable, enterprise-grade applications. 4. Get developer assistance customized to your organization code with Gemini Join Gemini product managers and Capgemini to learn how to customize Duet AI with your own private codebase and bring AI code assistance to your development teams. Also be sure to check out the Innovators Hive — a central hub where you can discover what’s new and next at Google Cloud — to dive into interactive demos and talk face-to-face with our developer experts. Here are a few of the demos you can expect to find in the Innovators Hive AppDev zone: Write your own Cloud Functions (with the help of Duet AI/Gemini) to interact with our programmable pinball machine that uses Pub/Sub to collect events from the game in real time. Learn all about serving a Gemma large language model using graphical processing units (GPUs) on Google Kubernetes Engine (GKE) Autopilot, working with the 2B and 7B parameter models in particular. There’s so much more to enjoy at Next ‘24 that you’ll have to see it to believe it. And if you can’t go in person, you can still register for a digital pass, and access keynotes, 100+ breakout sessions on demand. View the full article

The rise of data collaboration and use of external data sources highlights the need for robust privacy and compliance measures. In this evolving data ecosystem, businesses are turning to clean rooms to share data in low-trust environments. Clean rooms enable secure analysis of sensitive data assets, allowing organizations to unlock insights without compromising on privacy. To facilitate this type of data collaboration, we launched the preview of data clean rooms last year. Today, we are excited to announce that BigQuery data clean rooms is now generally available. Backed by BigQuery, customers can now share data in place with analysis rules to protect the underlying data. This launch includes a streamlined data contributor and subscriber experience in the Google Cloud console, as well as highly requested capabilities such as: Join restrictions: Limits the joins that can be on specific columns for data shared in a clean room, preventing unintended or unauthorized connections between data. Differential privacy analysis rule: Enforces that all queries on your shared data use differential privacy with the parameters that you specify. The privacy budget that you specify also prevents further queries on that data when the budget is exhausted. List overlap analysis rule: Restricts the output to only display the intersecting rows between two or more views joined in a query. Usage metrics on views: Data owners or contributors see aggregated metrics on the views and tables shared in a clean room. Using data clean rooms in BigQuery does not require creating copies of or moving sensitive data. Instead, the data can be shared directly from your BigQuery project and you remain in full control. Any updates you make to your shared data are reflected in the clean room in real-time, ensuring everyone is working with the most current data. Create and deploy clean rooms in BigQuery BigQuery data clean rooms are available in all BigQuery regions. You can set up a clean room environment using the Google Cloud console or using APIs. During this process, you set permissions and invite collaborators within or outside organizational boundaries to contribute or subscribe to the data. Enforce analysis rules to protect underlying data When sharing data into a clean room, you can configure analysis rules to protect the underlying data and determine how the data can be analyzed. BigQuery data clean rooms support multiple analysis rules including aggregation, differential privacy, list overlap, and join restrictions. The new user experience within Cloud console lets data contributors configure these rules without needing to use SQL. Lastly, by default, a clean room employs restricted egress to prevent subscribers from exporting or copying the underlying data. However, data contributors can choose to allow the export and copying of query results for specific use cases, such as activation. Monitor usage and stay in control of your data The data owner or contributor is always in control of their respective data in a clean room. At any time, a data contributor can revoke access to their data. Additionally, as the clean room owner, you can adjust access using subscription management or privacy budgets to prevent subscribers from performing further analysis. Additionally, data contributors receive aggregated logs and metrics, giving them insights into how their data is being used within the clean room. This promotes both transparency and a clearer understanding of the collaborative process. What BigQuery data clean room customers are saying Customers across all industries are already seeing tremendous success with BigQuery data clean rooms. Here’s what some of our early adopters and partners had to say: “With BigQuery data clean rooms, we are now able to share and monetize more impactful data with our partners while maintaining our customers' and strategic data protection.” - Guillaume Blaquiere, Group Data Architect, Carrefour “Data clean rooms in BigQuery is a real accelerator for L'Oréal to be able to share, consume, and manage data in a secure and sustainable way with our partners.” - Antoine Castex, Enterprise Data Architect, L’Oréal “BigQuery data clean rooms equip marketing teams with a powerful tool for advancing privacy-focused data collaboration and advanced analytics in the face of growing signal loss. LiveRamp and Habu, which independently were each early partners of BigQuery data clean rooms, are excited to build on top of this foundation with our combined interoperable solutions: a powerful application layer, powered by Habu, accelerates the speed to value for technical and business users alike, while cloud-native identity, powered by RampID in Google Cloud, maximizes data fidelity and ecosystem connectivity for all collaborators. With BigQuery data clean rooms, enterprises will be empowered to drive more strategic decisions with actionable, data-driven insights.” - Roopak Gupta, VP of Engineering, LiveRamp “In today’s marketing landscape, where resources are limited and the ecosystem is fragmented, solutions like the data clean room we are building with Google Cloud can help reduce friction for our clients. This collaborative clean room ensures privacy and security while allowing Stagwell to integrate our proprietary data to create custom audiences across our product and service offerings in the Stagwell Marketing Cloud. With the continued partnership of Google Cloud, we can offer our clients integrated Media Studio solutions that connect brands with relevant audiences, improving customer journeys and making media spend more efficient.” - Mansoor Basha, Chief Technology Officer, Stagwell Marketing Cloud “We are extremely excited about the General Availability announcement of BigQuery data clean rooms. It's been great collaborating with Google Cloud on this initiative and it is great to see it come to market.. This release enables production-grade secure data collaboration for the media and advertising industry, unlocking more interoperable planning, activation and measurement use cases for our ecosystem.” - Bosko Milekic, Chief Product Officer, Optable Next steps Whether you're an advertiser trying to optimize your advertising effectiveness with a publisher, or a retailer improving your promotional strategy with a CPG, BigQuery data clean rooms can help. Get started today by using this guide, starting a free trial with BigQuery, or contacting the Google Cloud sales team. View the full article

We are excited to announce that differential privacy enforcement with privacy budgeting is now available in BigQuery data clean rooms to help organizations prevent data from being reidentified when it is shared. Differential privacy is an anonymization technique that limits the personal information that is revealed in a query output. Differential privacy is considered to be one of the strongest privacy protections that exists today because it: is provably private supports multiple differentially private queries on the same dataset can be applied to many data types Differential privacy is used by advertisers, healthcare companies, and education companies to perform analysis without exposing individual records. It is also used by public sector organizations that comply with the General Data Protection Regulation (GDPR), the Health Insurance Portability and Accountability Act (HIPAA), the Family Educational Rights and Privacy Act (FERPA), and the California Consumer Privacy Act (CCPA). What can I do with differential privacy? With differential privacy, you can: protect individual records from re-identification without moving or copying your data protect against privacy leak and re-identification use one of the anonymization standards most favored by regulators BigQuery customers can use differential privacy to: share data in BigQuery data clean rooms while preserving privacy anonymize query results on AWS and Azure data with BigQuery Omni share anonymized results with Apache Spark stored procedures and Dataform pipelines so they can be consumed by other applications enhance differential privacy implementations with technology from Google Cloud partners Gretel.ai and Tumult Analytics call frameworks like PipelineDP.io So what is BigQuery differential privacy exactly? BigQuery differential privacy is three capabilities: Differential privacy in GoogleSQL – You can use differential privacy aggregate functions directly in GoogleSQL Differential privacy enforcement in BigQuery data clean rooms – You can apply a differential privacy analysis rule to enforce that all queries on your shared data use differential privacy in GoogleSQL with the parameters that you specify Parameter-driven privacy budgeting in BigQuery data clean rooms – When you apply a differential privacy analysis rule, you also set a privacy budget to limit the data that is revealed when your shared data is queried. BigQuery uses parameter-driven privacy budgeting to give you more granular control over your data than query thresholds do and to prevent further queries on that data when the budget is exhausted. BigQuery differential privacy enforcement in action Here’s how to enable the differential privacy analysis rule and configure a privacy budget when you add data to a BigQuery data clean room. Subscribers of that clean room must then use differential privacy to query your shared data. Subscribers of that clean room cannot query your shared data once the privacy budget is exhausted. Get started with BigQuery differential privacy BigQuery differential privacy is configured when a data owner or contributor shares data in a BigQuery data clean room. A data owner or contributor can share data using any compute pricing model and does not incur compute charges when a subscriber queries that data. Subscribers of a data clean room incur compute charges when querying shared data that is protected with a differential privacy analysis rule. Those subscribers are required to use on-demand pricing (charged per TB) or the Enterprise Plus edition (charged per slot hour). Create a clean room where all queries are protected with differential privacy today and let us know where you need help. Related Article Privacy-preserving data sharing now generally available with BigQuery data clean rooms Now GA, BigQuery data clean rooms has a new data contributor and subscriber experience, join restrictions, new analysis rules, usage metr... Read Article View the full article

Building all the tools you need from scratch or having complex installments for a cloud environment is a thing of the past now. In this fast-paced world, we need solutions that save time and make us more efficient. That’s where a one-stop for discovering, deploying, and managing all the essential tools you need comes into […]View the full article

Experience the magic at Google Cloud Next '24, where access to new opportunities awaits, helping you create, innovate, and supercharge your startup's growth. In the Startup Lounge, you’ll be able to grow your network, receive practical training on launching and scaling more efficiently, and acquire essential strategies to propel your startup forward. Over three days, dive into exclusive startup sessions and founder stories tailored to a startup's practical needs and transformational goals. You’ll also have the chance to learn the secrets to securing funding from Menlo Ventures, understand why AI unicorns are increasingly choosing Google Cloud and find out about the state of the startup landscape around the world. This year’s agenda is packed with exciting moments you won’t want to miss. Come listen to Dario Amodei, CEO and Co-Founder of Anthropic, and Elad Gil, a renowned entrepreneur and investor, as they discuss the transformative impact of generative AI on the business landscape in a fireside chat. You’ll also want to bookmark sessions from the Founder’s Series, which will offer insights on redefining strategies related to cost, productivity, and innovation from the founders of startups like LangChain, Hugging Face, Runway, Character.AI, AI21 Labs, Contextual AI, and more. These are only some of the can’t-miss sessions geared towards startup leaders — make sure you explore the full list here. But that’s not all, here’s what else startups can look forward to at Next ‘24. Build meaningful connections in the Startup Lounge Make sure to join us after the Main Keynote for the "Startups at Next” Kickoff Event, where we’ll be sharing some big announcements that will create new opportunities for you and your startup to grow with AI. Connect with peers, venture capitalists, and industry leaders during events such as the Xoogler Networking Event, the DEI Founder’s Ice Cream Social, and the Founder's Toast. Access tailored expertise with personal Office Hours Facing challenges or seeking to refine your strategy? Book direct access to Google experts and engineers in 15-minute Startup Expert sessions or pre-book 30-minute Office Hours sessions. These opportunities allow you to address technical questions, strategize on business development, or delve into AI, equipping you with the tools to unlock your startup's full potential. Book today before they are gone! Experience innovation at the Startup Showcases The excitement at Google Cloud Next '24 goes beyond traditional formats with the Startup Lounge featuring unique activations that foster creativity and growth. Witness over 25 startups launching new products or features on stage, enjoy a life-sized game of Chutes & Ladders by Product Science, and explore the future with Octo.ai in the Google Next Yearbook. Engage with AssemblyAI Speech-to-text AI for a unique fortune-telling experience and participate in the Suggestic three-day wellness challenge as you navigate Next ‘24. Don't miss out on grand prizes from each showcase and more. Apply for Cloud Credits and More with 1:1 Application Help This year, dedicated staff will be available at Next '24 to guide you through the Google for Startups Cloud Program application process. By joining, you'll gain access to startup experts, coverage for your Google Cloud and Firebase costs up to $200,000 USD (up to $350,000 USD for AI startups) over two years, technical training, business support, and exclusive Google-wide offers. We hope to see you at the event, and look forward to welcoming startup leaders from around the globe to Mandalay Bay in Las Vegas. View the full article

Google Kubernetes Engine (GKE) offers two different ways to perform service discovery and DNS resolution: the in-cluster kube-dns functionality, and GCP managed Cloud DNS. Either approach can be combined with the performance-enhancing NodeLocal DNSCache add-on. New GKE Autopilot clusters use Cloud DNS as a fully managed DNS solution for your GKE Autopilot clusters without any configuration required on your part. But for GKE Standard clusters, you have the following DNS provider choices: Kube-dns (default) Cloud DNS - configured for either cluster-scope or VPC scope, and Install and run your own DNS (like Core DNS) In this blog, we break down the differences between the DNS providers for your GKE Standard clusters, and guide you to the best solution for your specific situation. Kube-DNS kube-dns is the default DNS provider for Standard GKE clusters, providing DNS resolution for services and pods within the cluster. If you select this option, GKE deploys the necessary kube-dns components such as Kube-dns pods, Kube-dns-autoscaler, Kube-dns configmap and Kube-dns service in the kube-system namespace. kube-dns is the default DNS provider for GKE Standard clusters and the only DNS provider for Autopilot clusters running versions earlier than 1.25.9-gke.400 and 1.26.4-gke.500. Kube-dns is a suitable solution for workloads with moderate DNS query volumes that have stringent DNS resolution latency requirements (e.g. under ~2-4ms). Kube-dns is able to provide low latency DNS resolution for all DNS queries as all the DNS resolutions are performed within the cluster. If you notice DNS timeouts or failed DNS resolutions for bursty workload traffic patterns when using kube-dns, consider scaling the number of kube-dns pods, and enabling NodeLocal DNS cache for the cluster. You can scale the number of kube-dns pods beforehand using Kube-dns autoscaler, and manually tuning it to the cluster's DNS traffic patterns. Using kube-dns along with Nodelocal DNS cache (discussed below) also reduces overhead on the kube-dns pods for DNS resolution of external services. While scaling up kube-dns and using NodeLocal DNS Cache(NLD) helps in the short term, it does not guarantee reliable DNS resolution during sudden traffic spikes. Hence migrating to Cloud DNS provides a more robust and long-term solution for improved reliability of DNS resolution consistently across varying DNS query volumes. You can update the DNS provider for your existing GKE Standard from kube-dns to Cloud DNS without requiring to re-create your existing cluster. For logging the DNS queries when using kube-dns, there is manual effort required in creating a new kube-dns debug pod with log-queries enabled. Cloud DNS Cloud DNS is a Google-managed service that is designed for high scalability and availability. In addition, Cloud DNS elastically scales to adapt to your DNS query volume, providing consistent and reliable DNS query resolution regardless of traffic volume. Cloud DNS simplifies your operations and minimizes operational overhead since it is a Google managed service and does not require you to maintain any additional infrastructure. Cloud DNS supports dns resolutions across the entire VPC, which is something not currently possible with kube-dns. Also, while using Multi Cluster Services (MCS) in GKE, Cloud DNS provides DNS resolution for services across your fleet of clusters. Unlike kube-dns, Google Cloud’s hosted DNS service Cloud DNS provides Pod and Service DNS resolution that auto-scales and offers a 100% service-level agreement, reducing DNS timeouts and providing consistent DNS resolution latency for heavy DNS workloads. Cloud DNS also integrates with Cloud Monitoring, giving you greater visibility into DNS queries for enhanced troubleshooting and analysis. The Cloud DNS controller automatically provisions DNS records for pods and services in Cloud DNS for ClusterIP, headless and external name services. You can configure Cloud DNS to provide GKE DNS resolution in either VPC or Cluster (the default) scope. With VPC scope, the DNS records are resolvable with the entire VPC. This is achieved with the private DNS zone that gets created automatically. With Cluster scope, the DNS records are resolvable only within the cluster. While Cloud DNS offers enhanced features, it does come with usage-based costs. You save on compute costs and overhead by removing kube-dns pods when using Cloud DNS. Considering the typical cluster size workload traffic patterns, Cloud DNS is usually more cost effective than running kube-dns You can migrate clusters from kube-dns to Cloud DNS cluster scope without downtime or changes to your applications. The reverse (migrating from Cloud DNS to kube-dns) is not a seamless operation. NodeLocal DNSCache NodeLocal DNSCache is a GKE add-on that you can run in addition to kube-dns and Cloud DNS. The node-local-dns pod gets deployed on the GKE nodes after the option has been enabled (subject to a node upgrade procedure). Nodelocal DNS Cache (NLD) helps to reduce the average DNS resolution times by resolving the DNS requests locally on the same nodes as the pods, and only forwards requests that it cannot resolve to the other DNS servers in the cluster. This is a great fit for clusters that have heavy internal DNS query loads. Enable NLD during maintenance windows. Please note that node pools must be re-created for this change to take effect. Final thoughts The choice of DNS provider for your GKE Standard cluster has implications for the performance and reliability, in addition to your operations and overall service discovery architecture. Hence, it is crucial for GKE Standard users to understand their DNS options taking into account their application and architecture objectives. Standard GKE clusters allow you to use either kube-dns or Cloud DNS as your DNS provider, allowing you to optimize for either low latency DNS resolution or a simple, scalable and reliable DNS solution for GKE Standard clusters. You can learn more about DNS for your GKE cluster from the GKE documentation . If you have any further questions, feel free to contact us. We thank the Google Cloud team member who contributed to the blog: Selin Goksu, Technical Solutions Developer, Google View the full article

We're excited to announce a major enhancement to Google Cloud Backup and DR, making it simpler than ever to safeguard your critical Google Compute Engine virtual machines (VMs). You can now leverage the power of Google Cloud tags, including inheritance, to easily configure backup policies for Compute Engine VMs, ensuring consistent protection of your dynamic cloud environments. The challenge of managing VM backups Managing backups for a large number of Compute Engine VMs can be complex, especially when VMs are frequently added, removed, or modified. Manually assigning backup policies to individual VMs is time-consuming and can be error-prone, potentially leaving vital resources unprotected. The power of tag-based backups With our new tag-based backup approach, you can leverage the organizational power of tags to automate the protection of your Compute Engine VMs. Here's how it works: Assign your tags: Apply meaningful tags to your organization, folders, projects, or directly to your VMs. These tags might reflect application names, environments (production, development, testing), or criticality levels. Create tag-based policies: Within Google Backup and DR, you can create backup policies with your desired backup schedules, retention periods, and recovery regions. After policy creation, you can assign them to specific tags. Automate protection: Any VM with a matching tag is automatically assigned the backup policy. New VMs inheriting those tags immediately gain protection. Benefits of tag-based backups Simplified management: Drastically reduce the administrative overhead of configuring VM backups at scale by updating the policy attached to the tag rather than at an individual VM backup level. Consistent protection: Ensure new VMs with inherited tags from the project, folder, or organization are automatically protected without manual intervention. Flexibility: Adjust your backup strategies easily by modifying tags or creating new tag-based policies, including support for tags assigned using Terraform For example, you can easily tag all production projects with the tag backupdr-dynamicprotect:production. You could then create a backup policy that does the following: Takes daily snapshots of VMs in those projects that inherit the backupdr-dynamicprotect:production tag Retains backups for 30 days Updates your Terraform script to include your new protection tag to ensure protection of every new Compute Engine instance that’s created Getting started Review the Google Backup and DR documentation on tag-based backups. Develop a tagging strategy for your Compute Engine VMs. Create backup policies that target your chosen tags. Attend ARC205, Protect your critical workloads and recover your most critical asset - your data at Google Cloud Next. View the full article

Google Cloud credentials, including certifications and skill badges, empower you to validate your expertise in the ever-evolving landscape of cloud technology, and work towards that next promotion or prepare for a new career opportunity in cloud. Going to Google Cloud Next in Las Vegas April 9-11? Check out all the opportunities to lead with learning and earn credentials. It’s still not too late to register for a digital pass. According to a recent study led by PearsonVue, 92% of individuals with digital credentials are more confident in their abilities, and 81% have more confidence to explore new job opportunities. Whether you're looking to enhance your career prospects, transition into a new role, or showcase your cloud expertise, Google Cloud credentials offer a great pathway to elevate your career. We’re excited to introduce new, easy-to-use features that help to showcase your validated skills and demonstrate your expertise with Google Cloud. Our credential earners can now manage and share their credentials in one, easy-to-access place with the Google Cloud Credential Wallet, powered by Credly. We invite credential earners to opt-into the Google Cloud Skills Directory, to help you gain visibility to career-relevant cloud opportunities. Earners can take advantage of Labor Market Insights and discover open job opportunities based on verified skills and credentials, including job titles, locations, employers, and salary insights. Join us for an webinar on April 17 at 8am PT to discover how to make the most of these tools and features to showcase your skills and connect to employment opportunities. And don’t miss Google Cloud Next 2024 where we have a huge number of opportunities to lean into learning and earning a Google Cloud credential. Check out all the ways to learn at Next here. Meet Nikiya: A cloud skills success story Introducing Nikiya Simpson, a mom of three out of Little Rock Arkansas, who wears many hats: She’s a full stack engineer at a technology-based healthcare company and is pursuing her master’s degree in AI and Human-Computer Interaction at Georgia Tech, all while managing an active household. Nikiya, an avid learner on Google Cloud Skills Boost, tackles cloud challenges at work by utilizing on-demand learning labs during breaks. These labs provide her with hands-on experience with Google Cloud products and concepts related to Kubernetes, data processing, and application development. By acquiring targeted, validated skills with skill badges, Nikiya significantly enhanced her resume and career. “I was able to directly apply the skills obtained by earning skill badges in my career as a software engineer. I know more about cloud infrastructure, logging and monitoring, and how to create new solutions in ways that make me a better engineer. I’m also able to help people in my community spin up new projects, as well as explore new project ideas of my own.” Now, Nikiya continues to earn Google Cloud skill badges to advance her skillset, and even achieved the Associate Cloud Engineer certification, giving her the industry credibility needed to take on more complex projects at work. She even mentors aspiring cloud developers as an ambassador within the Women Techmakers program. Discover how Google Cloud Skills Boost and Google Cloud credentials fueled Nikiya's engineering journey. Check out her story here: Start your cloud skilling journey by earning a skill badge today Try earning a Google Cloud skill badge today. Whether you’re just getting started with AI, or an experienced technical practitioner, there’s skilling for every level. [Introductory] Prompt Design in Vertex AI [Introductory] Insights from Data with BigQuery [Intermediate] Build and Deploy ML Solutions on Vertex AI [Intermediate] Deploy to Kubernetes in Google Cloud See you at the upcoming webinar. In the meantime, make sure to: Create your Credly account. Claim your digital badges. Access your Credential Wallet to share your badges with your network and to your resume! View the full article

Google Cloud Next, our global exhibition of inspiration, innovation, and education, is coming to the Mandalay Bay Convention Center in Las Vegas starting Tuesday, April 9. We’re featuring a packed Security Professionals track, including more than 40 breakout sessions, demos, and lightning talks where you can discover new ways to secure your cloud workloads, get the latest in threat intelligence, and learn how AI is evolving security practices. Learn from an exciting lineup of speakers, including experts from Google Cloud and leaders from Charles Schwab, L’Oreal, Snap, Fiserv, Spotify, and Electronic Arts. Register here and get ready to defend your business with the latest in frontline intelligence, intel-driven SecOps, and cloud innovation. Here are some must-attend session for security professionals: Security keynote What’s next for security professionals Organizations need powerful security solutions to protect their most sensitive assets, but the familiar approach of bolting on more security products is unsustainable. Instead, the convergence of key capabilities can reduce complexity, increase productivity, and deliver stronger security outcomes. In this spotlight, leaders from Google and Mandiant will share our insight into the threat landscape and our latest innovations that make Google part of your security team. Featured speakers: Sunil Potti, vice president and general manager, Cloud Security, Google Cloud Kevin Mandia, CEO Mandiant, Google Cloud Parisa Tabriz, vice president, Chrome Browser, Google Steph Hay, director of User Experience, Cloud Security, Google Cloud Security breakout sessions How AI can transform your approach to security Generative AI from Google Cloud can help seasoned security professionals quickly discover, investigate, and respond to threats, and help newcomers boost their skills to get more things done, faster. Join this session to learn about new features, hear from customers, and gain insights on how to address common security challenges with AI. Featured speakers: Anton Chuvakin, senior staff security consultant, Google Cloud Bashar Abouseido, senior vice president and CISO, Charles Schwab Steph Hay, director of User Experience, Cloud Security, Google Cloud Simplified Google Cloud network security: Zero Trust and beyond This demo-packed session on how to proactively defend your cloud network with Google Cloud's machine learning-powered Zero Trust architecture reveals the latest Cloud next-generation firewall (NGFW), Secure Web Proxy and Cloud Armor enhancements, partner integrations, and real-world use cases. Learn to neutralize unauthorized access, sensitive data loss, and other cloud attack vectors across public and hybrid cloud environments. Featured speakers: Ashok Narayanan, principal engineer, Google Cloud Tobias Pischl, director of product management, Broadcom, Inc. Olivier Richaud, vice president, Platforms, Symphony Sid Shibiraj, group product manager, Google Cloud Start secure, stay secure: Build a strong security foundation for your cloud In this session, you’ll learn how to start secure and stay secure in the cloud using the Security Foundation, which provides recommended controls to common cloud adoption use cases, including infrastructure modernization, AI workloads, data analytics, and application modernization. Featured speakers: William Bear, cloud architect, Mayo Clinic Dhananjay Dalave, strategic cloud engineer, Google Cloud Marcus Hartwig, product marketing lead, Google Cloud Stacy Moening, director of product management - IT, Mayo Clinic Anil Nandigam, product marketing lead, Google Cloud Practical use cases for AI in security operations AI isn’t a future concept: It’s here and available, with early user feedback showing that AI can reduce the time required for common analyst tasks such as triaging complex cases by seven times. In this session, we’ll dive into the real-world applications of AI in security operations with hands-on demonstrations and case studies. Featured speakers: Payal Chakravarty, product manager, Google Cloud Mark Ruiz, head of Cybersecurity Analytics, Pfizer A cybersecurity expert's guide to securing AI products with Google SAIF AI is rapidly advancing, and it’s important that effective security and privacy strategies evolve along with it. To support this evolution, Google Cloud introduced the Secure AI Framework (SAIF). Join us to learn why SAIF offers a practical approach to addressing the concerns that are top of mind for security and privacy professionals like you, and how to implement it. Featured speakers: Anton Chuvakin, senior staff security consultant, Google Cloud Shan Rao, group product manager, Google Cloud Preventing data exfiltration with Google Cloud’s built-in controls Join us to hear from Charles Schwab and Commerzbank on their security journey to mitigate data exfiltration risks and reduce the rising cost of a breach. You’ll also learn more about the Google Cloud platform security technologies, including VPC Service Controls and Organization Restrictions, and how they can be deployed to effectively address different threat vectors. Featured speakers: Sri B, product manager, Google Cloud Steffen Kreis, senior cloud architect, Commerzbank AG Florin Stingaciu, product manager, Google Cloud Jonathan Wentz, cloud architect, Charles Schwab Go beyond multi-cloud CNAPP security with Security Command Center Enterprise Learn how to implement a full life cycle approach to reduce cloud risks with the industry’s first solution that converges best-in-class cloud security and enterprise security operations capabilities. Hear how Verizon security teams assess and mitigate cloud risks, and how your organization can proactively manage threats, posture, data, and identities, and respond to security events across clouds with a single, converged solution operating at Google Cloud scale. Featured speakers: Arshad Aziz, director, Cloud Security, Verizon Connor Hammersmith, Security Command Center GTM, Google Cloud Jess Leroy, director, product management, Google Cloud Cloud compromises: Lessons learned from Mandiant investigations in 2023 This session will draw from real-world experiences and highlight the top threats that emerged in 2023 in relation to cloud security. Participants will gain valuable insights through case studies derived from Mandiant's incident response engagements in 2023, covering numerous threat actors such as Scattered Spider aka UNC3944. Featured speakers: Omar ElAhdan, principal consultant, Mandiant, Google Cloud Will Silverstone, senior consultant, Google Cloud Learn to love IAM: The most important step in securing your cloud infrastructure Join this session to learn all things to get started with IAM, including how to structure and set up identities, and access resources with minimal risk to your cloud environment. Featured speakers: Elliott Abraham, cloud security architect, Google Cloud Michele Chubirka, cloud security advocate, Google Cloud Your model. Your data. Your business: A privacy-first approach to generative AI Want to learn how to quickly and easily train and fine-tune production AI models? Use your data and own your model? Train in the cloud or on-premises, pain-free? Then this is the session for you! Featured speakers: Todd Moore, vice president, Encryption Solutions, Thales Amit Patil, director, Cloud Security Platform and Products, Google Cloud Nelly Porter, director, product management, Google Cloud Secure enterprise browsing: New ways to strengthen endpoint security Discover how to leverage Chrome Enterprise to apply built-in security measures, including data-loss prevention and context-aware access controls, plus the latest product announcements. You’ll leave with actionable best practices, and tips to safeguard your corporate data and empower your team. Featured speakers: Mark Berschadski, director, product management, Chrome Browser Enterprise, Google Cloud Tanisha Rai, Google Cloud Security product manager, Google Using Security Command Center for multi-cloud threat detection and response Learn how Google detects threats against Google Cloud to protect multicloud environments and find out how industry-leading Mandiant Threat Intelligence combines with planet-scale data analytics to detect new and persistent threats to cloud environments, cloud identities, and cloud workloads. Featured speakers: Ken Duffy, staff cloud architect, Telus Corporation Timothy Peacock, senior product manager, Google Cloud Jason Sloderbeck, group product manager, Security Command Center, Google Cloud TD shows Google Cloud’s shared fate model in action Discover how expecting more from Google Cloud’s shared fate model has helped TD accelerate its transformation with peace of mind. Join this session to learn how TD is building out a generative AI platform designed to expand its business, create a next generation user experience, and enhance the bank’s security posture. Featured speakers: Dan Bosman, senior vice president and CIO, TD Securities and TBSM, TD Securities Andy Chang, group product manager, Google Cloud Sean Leighton, principal architect, Google Cloud Address retail security challenges with AI Retail organizations are battling some challenging, financially motivated threat actors. To keep pace and stay within your resources budget, AI may provide some answers. In this session, we’ll discuss the evolving retail cyber-threat landscape and explore how AI can help organizations improve their security. Featured speakers: Brian Klenke, CISO, Bath & Body Works Luke McNamara, Deputy Chief Analyst, Mandiant Intelligence Harpreet Sidhu, North American Cybersecurity Lead, Accenture To explore the full catalog of breakout sessions and labs designed for security professionals, check out the security track in the Next ‘24 catalog. View the full article