Delivery engineering = GitOps at State Farm

[weaveworks]

Mae Large is Architecture Manager at State Farm, the largest provider of auto and home insurance in the United States. State Farm is ranked No. 36 on the 2020 Fortune 500 list of largest companies. With over 200 unique products using GitOps and 1,800 software developers, State Farm operates at an elite level when it comes to software delivery. Large and her team are always looking to push the limits to find new ways to optimize their operations and deliver software better, and faster. She has found GitOps to be one of the key drivers of this innovation at scale. Let's dig into Large's insights on GitOps and software delivery at one of the largest insurance providers in the United States.

Delivery engineering = GitOps

As the owner of the GitOps Platform team, Large is responsible for “delivery engineering”. She explains that delivery engineering is about enabling every product team to deploy changes to platforms in a simple, compliant, and repeatable manner - a very succinct explanation of the term "GitOps".

Our goal is to give every product team the ability to deploy their changes to our modern strategic platforms in a simple, compliant and repeatable manner.

They achieved their primary goal to improve the developer experience by enabling a self-service developer platform. The job of the GitOps platform team in this case is to enforce compliance while giving more power to developers. This is a challenging balance to walk, but Large and her team at State Farm have found a way to achieve this balance.

State Farm's GitOps Adoption

Initial adoption of GitOps began with a pitch from one of the product teams in February 2019. The first deployment took place only six months later.

2020 was the year of adoption of GitOps.

• Jan 2020 - General availability of GitOps (Public Cloud and Kubernetes on-premises)

• Dec 2020 - GitOps for on-premises Cloud Foundry

• Jan 2021 - GitOps for Kubernetes on-premises. Moved k8s production namespaces under Flux multitenancy

What's interesting from this timeline is that GitOps adoption took place in phases, and wasn't done over a single quarter. This ensured adequate time for the various teams to get on board at their own pace. It also provided the GitOps product team time to adapt to issues as they arose, ensuring a smoother transition.

The team closely monitored GitOps adoption metrics such as the number of config groups, and config repos being used. They also track which parts of their GitOps cookbook is being used by developers as they adopt GitOps and correlate this with feedback they get on Rocket.Chat, their internal developer chat platform.

The goal for 2021 is to make the value of GitOps more visible across the organization. They specifically want to see improvements in visible deployment frequency and change lead time data.

The key elements of GitOps

Large lists three elements that help drive successful delivery engineering, or GitOps.

1. Risk-aware decisions: This is done by enforcing small frequent changes that make it to production quickly. This leads to an awareness of what lines of code were changed, what kind of validation was done prior to the merge request, and being aware of every phase of the pipeline end-to-end. This is especially key to State Farm as it operates in a highly regulated industry.

2. Developer-friendly: GitOps enables developers to collaborate better, and developers love it. To take advantage of this Large's team was intentional about creating a community and ecosystem around GitOps. This involved a combination of IDEs and Git promoting tooling that developers already use day to day, and Rocket.Chat for communication.

3. Compliant: GitOps brought separation of duties across teams. Additionally, it presented a full audit trail of all events end-to-end. The best part is that all of this is done automatically.

Further commenting on compliance, Large explains how her team has had to partner with the change management team. They put checks in place so that no one person can make a change without oversight. Every single change goes through a merge request process. There are no direct pushes to production.

How to get started with GitOps

Large has a few words of advice for those getting started with GitOps.

1. Describe everything as code: She recommends that if anything can be described as code, it should be. She advises to make everything declarative. This is the first step to GitOps adoption.

You always have to factor in scale. Describing things as code, specifically embracing the declarative nature of a lot of these modern platforms is such a key and powerful enabler.

2. Know your customers: In this case, the customers are the developers who build products running in different platforms. With GitOps, developers should not have to do anything additional for compliance. Compliance is built-in.

3. A 'No' doesn't mean 'Stop': In your GitOps adoption journey there will be obstacles on the way. What's important is to always keep a pulse on what’s happening in the industry.

Mae Large is excited about Flux2 and the improvements it brings in terms of monitoring. She is looking forward to having Prometheus and Grafana built-in. She also has her sights set on what she likes to call 'GitOps+' which is about greater observability via GitOps. GitOps enables another state of reality apart from monitoring systems. She also talks about progressive delivery of applications, something that is becoming a larger part of GitOps discussions these days.

Mae Large, Architecture Manager at State Farm

I'm currently an Architecture Manager at State Farm. I've worked as a software developer throughout my career starting in the Philippines where I was born and raised. I had the opportunity to work in Finland and in India at a consulting capacity before joining State Farm in 2006 first as a consultant, and a permanent employee in 2010. I own and drive the Delivery Engineering topic at State Farm where I continuously simplify how code change ultimately makes it to the hands of our customers.

I recently earned the following industry certifications:

• AWS Solutions Architect Associate

• AWS Developer Associate

• AWS SysOps Associate

• AWS Solutions Architect Professional

• AWS DevOps Engineer Professional

• CKAD (Kubernetes Application Developer)

• HashiCorp - Terraform, Vault and Consul

I'm an avid supporter of 'women in technology' initiatives at State Farm by holding tech talks and mentoring women to advance their technical acumen. Outside of work, I spend time with my husband and our 2 kids and volunteer at our church

To be notified of future episodes:

Listen to the full episode:

Apple Podcasts | Spotify | Stitcher | SoundCloud | Pocket Cast | Google Podcast | Overcast

The Art of Modern Ops · Delivery engineering = GitOps at State Farm

View the full article