Multi-cluster Application Deployment Made Easy with GitOpsSets

[weaveworks]

GitOpsSets is one of our newest features released in the latest Weave GitOps Enterprise product launch. This feature enables Platform Operators to have a single resource definition for an application for multiple environments and a fleet of clusters. GitOpsSets reduces the complexity of managing multiple applications and resources, streamlines deployment processes, and ensures consistency across your infrastructure.

In this article, we will explore in depth our latest feature, why it’s needed, its benefits, and how you can get started.

The Problem: Cloud-Native Infrastructure Complexity

With the majority of organizations running numerous clusters hosting multiple applications, managing the configuration of these different environments is a nontrivial challenge faced by most DevOps teams. Let’s just take a moment to consider the complexity involved:

A typical microservices application deployed on a Kubernetes cluster has numerous Pods, and each of those Pods will be at different stages of their delivery pipeline being deployed across development, staging, and production environments. That’s already a lot of moving parts for just one application, and organizations have more than just one application.

The 2021 Datadog Container Use report shows that on average, an organization is running 1000 Pods, and the trend is rapidly rising. Therefore an average Kubernetes estate is running 20 clusters hosting 1000 Pods, with each Pod running at different versions as they are promoted through development, staging, and production. That’s a lot of configuration complexity to keep up with.

Git is the ideal tool to manage the complexity of all these different configurations. A previous blog post provided an overview of the many different ways to structure Git repositories for micro-service deployments across multiple Kubernetes environments. A real-world example of how Zscaler manages its multiple tenant deployments - here is the recorded talk from Zscaler.

Automating Complexity with Weave GitOpsSets

The pure GitOps approach does make it considerably easier to manage the complexity of deploying multiple applications across multiple clusters, however, there’s still quite a lot of manual work involved. Weave GitOps has automated most of the manual steps with numerous features - such as GitOpsSets - and wrapped it all up with a nice GUI. Let’s take a look.

What are GitOpsSets?

GitOpsSets enable Platform Operators to have a single definition for an application for multiple environments and a fleet of clusters. With a single resource definition, the Ops team can generate environment and cluster-specific configurations. GitOpsSets will create all the objects and Flux primitives that are required to deploy this application successfully. Now Operators no longer need to edit 100’s of files - all of that can now be done with a single command.

GitOpsSets combine Generators and Templates:

• Generators: these are dynamic functions that can fulfill a variety of tasks, they generate key/value maps of data that can be injected into templates.

• Templates: These are templated YAML files describing Kubernetes resources which are combined with the generated data and applied directly to the cluster.

GitOpsSets supports multiple generators including:

• List Generator: The simplest generator which provides a list of Key/Value pairs that you want to feed the template with.

• Git Generator: Enable to extract a set of files (environment-specific configurations) from a Flux GitRepository, and make the contents of these available to the templates, this would let you have config in app-dev.json, app-staging.json and app-production.json for example, and the contents of these would be available to the templates.

• Matrix Generator: Combine generators into the desired compounded input.

• Pull request Generator: Automatically discover open pull requests within a repository to generate a new deployment.

• API Client Generator: Poll an HTTP endpoint and parse the result as the generated values.

• Cluster Generator: Query for a set of Weave GitOps Clusters matching a set of labels.

View the full article