gitops Weave AI Controllers: GitOps Automation for Large Language Models (LLMs)

[weaveworks]

It’s with great excitement that we announce the release of the Weave AI controllers for Large Language Models (LLMs). Weave AI controllers ease the adoption of open-sourced LLMs like Llama-2, Mistral, Zephyr, and Falcon in enterprise environments through GitOps automation. We made it simple and efficient for Machine Learning teams to deploy, manage and fine-tune LLMs on any Kubernetes infrastructure while ensuring strong security and governance. Recent CNCF surveys show GitOps is the standard operating model for Kubernetes based workloads.

LLMs are on the rise but so are complexities in operating them 

The usage of LLMs has grown dramatically over the past several years and so has their rate of adoption. According to this survey, nearly one in ten (8.3%) machine learning teams have already deployed an LLM application into production and nearly half (43.3%) have production deployment plans within the next 12 months. However data privacy and the need to protect proprietary data are the largest roadblocks for production deployment. Many organizations are struggling with manual, ad-hoc methods of deploying downloaded LLMs that can lead to security risks and lack of governance and compliance. Versioning and updates as well as integration into existing infrastructure are also hurdles that need to be overcome.

Weaveworks addresses these challenges with GitOps workflows, Flux-based AI Controllers, and a signing and verification process that enhances security and compliance even in regulated industries. The Weave AI Controllers will be shipped beginning December 2023 with our standard subscriptions in Weave GitOps Assured and Enterprise.  

Streamline ops and free up development time

Weave AI controllers were designed to address two main use cases:

1. Enabling a self-service platform of AI models, tools and applications for developers, and 
2. Facilitating fine-tuning of models with sensitive data for enterprise-grade efficiency, security, and reliability. 

Many ML teams have been exploring these models and their capabilities in development or on small scale production clusters, with Weave AI Controllers teams can now move into enterprise scale with the necessary security guardrails and deploy to production quickly.

A Kubernetes based infrastructure and deployment pipeline using CRDs, YAMLs and GitOps can easily remedy most deployment challenges while provisioning monitoring and rollbacks. Weave GitOps and our AI controllers are leveraging cloud native technologies and the declarative management approach to build automated and streamlined workflows on prem, hybrid or in the cloud. We want data scientists to focus on the application and stop worrying about infrastructure tasks.

View the full article