Kaito Project: Kubernetes Operator for AI Toolchain Management

KAITO is an automation tool designed for AI/ML model inference and tuning in Kubernetes clusters, targeting popular open-source models like Falcon and Phi-3. Key features include managing large model files via container images, providing preset configurations, and supporting open-source inference runtimes. The upcoming KAITO v0.5.0 will introduce Retrieval-augmented Generation (RAG) support. The current version, v0.4.6, was released on May 14, 2025. KAITO utilizes the Kubernetes Custom Resource Definition (CRD) pattern, allowing users to define GPU requirements and workloads through a workspace resource. Key components include a workspace controller for automating deployment and a node provisioner for managing GPU nodes. Users can deploy models using simple commands and track workspace status through Kubernetes. There are guidelines for updating models, customizing parameters, and a focus on distinguishing between instruct and non-instruct models. KAITO encourages community contributions while adhering to a Contributor License Agreement.

Source link