Search the Community

Private connectivity is critical for organizations to secure data access and meet regulatory requirements — particularly if their applications access an operational database. Traditionally, organizations have relied on VPC Network Peering leveraging private services access, but that can be cumbersome at scale. And for more demanding environments, there’s Private Service Connect, which helps customers simplify and streamline private connectivity at scale, providing enhanced security controls, better IP space management, and flexible network topologies. This week at Google Cloud Next, we announced that Private Service Connect is now fully integrated with Cloud SQL, Google Cloud’s fully managed database service for PostgreSQL, MySQL, and SQL Server. With high availability (99.99% SLA), high performance, and rich data-protection capabilities, Cloud SQL is a popular choice for customers that run their business-critical applications on Compute Engine, Google Kubernetes Engine (GKE), Cloud Run, or on-premises. Integration with Private Service Connect makes it easier for these applications to securely connect to Cloud SQL without sacrificing on flexibility, scalability or security. Private Service Connect for Cloud SQL for SQL Server is now generally available, in addition to pre-existing support for MySQL and PostgreSQL. Benefits of Private Service Connect with Cloud SQL You can expect to find the following benefits from using Private Service Connect with Cloud SQL: Private Service Connect makes it easy to connect to Cloud SQL from multiple consumer VPC networks that belong to different groups, teams, projects, or organizations. Private Service Connect makes it easier to manage the IP space since you only need one IP address to access a Cloud SQL instance from the VPC. Because you choose this IP from your VPC, it’s like you access Cloud SQL as if it were a local target. Cloud SQL on Private Service Connect allows a single instance to manifest in different VPCs with different endpoints. This makes multi-tenanted deployments (e.g., SaaS applications) easier to manage and more secure. Private Service Connect simplifies connecting to primary instances, same-region read replicas, and cross-region read replicas in Cloud SQL. Configuring Private Service Connect with Cloud SQL Now let’s look into the details of configuring Private Service Connect with Cloud SQL: 1. Create a Cloud SQL instance Create a Cloud SQL Instance with Private Service Connect enabled. 2. Create a Private Service Connect endpoint Create a Private Service Connect Endpoint within your VPC to connect to Cloud SQL instance. You can choose an IP address for the endpoint from your VPC. While creating the endpoint, use the service attachment URI from the instance created in step #1. You can also allowlist specific projects within the VPC that can connect to Cloud SQL instance via Private Service Connect. 3. Connect to a Cloud SQL instance Once the Private Service Connect endpoint is created, you are ready to connect to the Cloud SQL instance. You can connect by using an internal IP address, a DNS record, the Cloud SQL Auth Proxy, or the Cloud SQL Language Connectors. Refer to the documentation for details of all the above steps along with code samples. Deployment patterns Now let’s look at different deployment patterns that Private Service Connect enables for Cloud SQL. 1. Connecting from single VPC to Cloud SQL If you have all your applications in a single VPC, then you can connect to the Cloud SQL instance using a single Private Service Connect endpoint within that VPC. If it’s a shared VPC, then you can control the list of projects that can connect. For example, let’s say you have both the production project and test project in the same VPC and you want to restrict access to production Cloud SQL instances to only the production project. You can do so by allowlisting only the production project. 2. Connecting from multiple VPCs to Cloud SQL If you have your applications in multiple VPCs and you want those applications to connect simultaneously to Cloud SQL, it is easy to do so using Private Service Connect. You just need to create a separate Private Service Connect endpoint within each VPC. 3. Connecting to Cloud SQL read replica instances In Cloud SQL you can create same-region read replicas to scale the read throughput and you can create cross-region read replicas for either disaster recovery or to reduce the read-latency for applications in different regions. It’s easier to connect privately to those read replicas using Private Service Connect. Since each read replica is considered a separate Cloud SQL instance, you need to create a new Private Service Connect endpoint for each read replica and connect to it from your applications. 4. Connecting from on-premises to Cloud SQL and connecting from a different region to Cloud SQL If an application hosted on-premises wants to connect to a Cloud SQL instance, then you can use Cloud Interconnect or VPN to reach the VPC that has the Private Service Connect endpoint for the Cloud SQL instance. You can also connect from an application in one region to the Private Service Connect endpoint in a different region within the same VPC using Private Service Connect global access. Benefits of Private Service Connect over Private Service Access Cloud SQL on Private Service Connect provides better security controls and ease of operations compared to traditional VPC network peering-based connectivity options such as Private Service Access. In Private Service Access, it’s cumbersome to have multiple VPCs simultaneously access a single Cloud SQL instance since you need to use VPN, Cloud Interconnect, or intermediate proxy (SOCKS5). Private Service Connect makes it easier since you can create separate Private Service Connect endpoints within each VPC. Private IP address space is limited and address space management is complex when using Private Service Access since you need to reserve an entire subnet range for Cloud SQL instances in each region. Private Service Connect brings a massive reduction in IP space usage, as you need to reserve only one IP address for each Private Service Connect endpoint that is mapped to the Cloud SQL instance. When using Private Service Access, you need to allowlist an entire subnet range beforehand when setting egress firewall rules, which can be overly permissive and an operational burden. With Private Service Connect, you can set granular firewall rules, which is better from a security perspective. When using Private Service Access, VPC Peering quotas can create challenges for deployment at scale. This is not an issue with Private Service Connect, as peering quotas don’t apply. Conclusion In conclusion, Private Service Connect with Cloud SQL offers strong security, ease of use, simplicity of operations, and seamless scalability. Click here for documentation on step-by-step instructions of using Private Service Connect with Cloud SQL. Click here to learn more about Cloud SQL. View the full article

Private Service Connect is a Cloud Networking offering that creates a private and secure connection from your VPC networks to a service producer, and is designed to help you consume services faster, protect your data, and simplify service management. However, like all complex networking setups, sometimes things don’t work as planned. In this post, you will find useful tips that can help you to tackle issues related to Private Service Connect, even before reaching out to Cloud Support. Introduction to Private Service Connect Before we get into the troubleshooting bits, let’s briefly discuss the basics of Private Service Connect. Understanding your setup is key for isolating the problem. Private Service Connect is similar to private services access, except that the service producer VPC network doesn't connect to your (consumer) network using VPC network peering. A Private Service Connect service producer can be Google, a third-party, or even yourself. When we talk about consumers and producers, it's important to understand what type of Private Service Connect is configured on the consumer side and what kind of managed service it intends to connect with on the producer side. Consumers are the ones who want the services, while producers are the ones who provide them. The various types of Private Service Connect configurations are: Private Service Connect endpoints are configured as forwarding rules which are allocated with an IP address and it is mapped to a managed service by targeting a Google API bundle or a service attachment. These managed services can be diverse, ranging from global Google APIs to Google Managed Services, third-party services, and even in-house, intra-organization services. When a consumer creates an endpoint that references a Google APIs bundle, the endpoint's IP address is a global internal IP address – the consumer picks an internal IP address that's outside all subnets of the consumer's VPC network and connected networks. When a consumer creates an endpoint that references a service attachment, the endpoint's IP address is a regional internal IP address in the consumer's VPC network – from a subnet in the same region as the service attachment. Private Service Connect backends are configured with a special Network Endpoint Group of the type Private Service Connect which refers to a locational Google API, or to a published service service attachment. A service attachment is your link to a compatible producer load balancer. And Private Service Connect interfaces, a special type of network interface that allows service producers to initiate connections to service consumers. How Private Service Connect works Network Address Translation (NAT) is the underlying network technology that powers up Private Service Connect using Google Cloud’s software-defined networking stack called Andromeda. Let's break down how Private Service Connect works to access a published service based on an internal network-passthrough load balancer using a connect endpoint. In this scenario, you set up a Private Service Connect endpoint on the consumer side by configuring a forwarding rule that targets a service attachment. This endpoint has an IP address within your VPC network. When a VM instance in the VPC network sends traffic to this endpoint, the host’s networking stack will apply client-side load balancing to send the traffic to a destination host based on the location, load and health.The packets are encapsulated and routed through Google Cloud’s network fabric.At the destination host, the packet processor will apply Source Network Address Translation (SNAT) and Destination Network Address Translation (DNAT) using the NAT subnet configured and the producer IP address of the service, respectively.The packet is delivered to the VM instance serving as the load balancer’s backend.All of this is orchestrated by Andromeda’s control plane; with a few exceptions, there are no middle box or intermediaries involved in this process, enabling you to achieve line rate performance. For additional details, see Private Service Connect architecture and performance. With this background, you should be already able to identify the main components where issues could occur: the source host, the network fabric, the destination host, and the control-plane. Know your troubleshooting toolsThe Google Cloud console provides you with the following tools to troubleshoot most of the Private Service Connect issues that you might encounter. Connectivity TestConnectivity Tests is a diagnostics tool that lets you check connectivity between network endpoints. It analyzes your configuration and, in some cases, performs live data-plane analysis between the endpoints. Configuration Analysis supports Private Service Connect: Consumers can check connectivity from their source systems to PSC endpoints (or consumer load balancers using PSC NEG backends), while producers can verify that their service is operational for consumers. Live Data Plane Analysis supports both Private Service Connect endpoints for published services and Google APIs: Verify reachability and latency between hosts by sending probe packets over the data plane. This feature provides baseline diagnostics of latency and packet loss. In cases where Live Data Plane Analysis is not available, consumers can coordinate with a service producer to collect simultaneous packet captures at the source and destination using tcpdump. Cloud Logging Cloud Logging is a fully managed service that allows you to store, search, analyze, monitor, and alert on logging data and events. Audit logs allow you to monitor Private Service Connect activity. Use them to track intentional or unintentional changes to Private Service Connect resources, find any errors or warnings and monitor changes in connection status for the endpoint. These are mostly useful when troubleshooting issues during the setup or updates in the configuration. In this example, you can track endpoint connection status changes (pscConnectionStatus) by examining audit logs for your GCE forwarding rule resource: code_block <ListValue: [StructValue([('code', 'resource.type="gce_forwarding_rule"\r\nprotoPayload.methodName="LogPscConnectionStatusUpdate"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3ef828a856a0>)])]> VPC Flow Logs to monitor Private Service Connect traffic. Consumers can enable VPC Flow Logs at the client subnet to monitor traffic flow directed to the Private Service Connect endpoint. This allows the consumer to validate traffic egressing the VM instance. Producers can enable VPC Flow Logs at the target load balancer subnet to monitor traffic ingressing their VM instances backends. Consider that VPC Flow Logs are sampled and may not capture short-lived connections. To get more detailed information, run a packet capture using tcpdump. Cloud Monitoring Another member of the observability stack, Cloud Monitoring can help you to gain visibility into the performance of Private Service Connect. Another member of the observability stack, Cloud Monitoring can help you to gain visibility into the performance of Private Service Connect. Producer metrics to monitor Published services. Take a look at the utilization of service attachment resources like NAT ports, connected forwarding rules and connections by service attachment ID to correlate with connectivity and performance issues. See if there are any dropped packets at the producer side (Preview feature). Received packets dropped count are related to NAT resource exhaustion. Sent packets dropped count indicate that a service backend is sending packets to a consumer after the NAT translation state has expired. When this occurs, make sure you are following the NAT subnets recommendations. A packet capture could bring more insights on the nature of the dropped packets. Using this MQL query, producers can monitor NAT subnet capacity for a specific service attachment: code_block <ListValue: [StructValue([('code', 'fetch gce_service_attachment\r\n| metric\r\n \'compute.googleapis.com/private_service_connect/producer/used_nat_ip_addresses\'\r\n| filter (resource.region == "us-central1"\r\n && resource.service_attachment_id == "[SERVICE_ATTACHMENT_ID]")\r\n| group_by 1m,\r\n [value_used_nat_ip_addresses_mean: mean(value.used_nat_ip_addresses)]\r\n| every 1m\r\n| group_by [resource.service_attachment_id],\r\n [value_used_nat_ip_addresses_mean_mean:\r\n mean(value_used_nat_ip_addresses_mean)]'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3ef828a856d0>)])]> Consumer metrics to monitor endpoints. You can track the number of connections created, opened and closed from clients to the Private Service Connect endpoint. If you see packet drops, take a look at the producer metrics as well. For more information, see Monitor Private Service Connect connections. TIP: Be proactive and set alerts to inform you when you are close to exhausting a known limit (including Private Service Connect quotas). In this example, you can use this MQL query to track PSC Internal LB Forwarding Rules quota usage. code_block <ListValue: [StructValue([('code', "fetch compute.googleapis.com/VpcNetwork\r\n| metric\r\n 'compute.googleapis.com/quota/psc_ilb_consumer_forwarding_rules_per_producer_vpc_network/usage'\r\n| group_by 1m, [value_usage_mean: mean(value.usage)]\r\n| every 1m\r\n| group_by [], [value_usage_mean_aggregate: aggregate(value_usage_mean)]"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x3ef828a85130>)])]> Read the manualConsult the Google Cloud documentation to learn about the limitations and supported configurations. Follow the Private Service Connect guides. Especially for new deployments, it is common to misconfigure a component or find that it is not compatible or supported yet. Ensure that you have gone through the right configuration steps, and go through the limitations and compatibility matrix.Take a look at the VPC Release notes. See if there are any known issues related to Private Service Connect, and look for any new features that could have introduced unwanted behavior. Common issuesSelecting the right tool depends on the specific situation you encounter and where you are in the life cycle of your Private Service Connect journey. Before you start, gather consumer and producer project details, and that in fact, this is a Private Service Connect issue, and not a Private services access problem. Generally, you can face issues during setup or update of any related component or additional capability, or the issues could be present during runtime, when everything is configured but you run into connectivity or performance issues. Issues during setupMake sure that you are following the configuration guide and you have an understanding of the scope and limitations. Check for any error message or warning in the Logs Explorer.Verify that the setup is compatible and supported as per the configuration guides.See if there is any related quota exceeded like the Private Service Connect forwarding rules.Confirm whether there is an organization policy that could prevent the configuration of Private Service Connect components.Issues during runtimeIsolate the issue to the consumer or the producer side of the connection. If you are on the consumer side, check if your endpoint or backend is accepted in the connection status at the Private Service Connect page. Otherwise, review in the producer side the accept/reject connection list and the connection reconciliation setup.If your endpoint is unreachable, check bypassing DNS resolution and run a Connectivity Test to validate routes and firewalls from the source endpoint IP address to the PSC endpoint as destination. On the service producer side, check if the producer service is reachable within the producer VPC network, and from an IP address in the Private Service Connect NAT subnet.If there is a performance issue like network latency or packet drops, check if Live Data Plane Analysis is available to determine a baseline and isolate an issue with the application or service. Also, check the Metrics Explorer for any connections or port exhaustion and packet drops.Working with Cloud SupportOnce that you have pinpointed the issue and you have analyzed the problem, you may need to reach out to Cloud Support for further assistance. To facilitate a smooth experience, be sure to explain your needs, clearly describe the business impact and give enough context with all the information collected. View the full article