Managing large, complex GPU clusters in data centers is a daunting task, requiring meticulous oversight of cooling, power, networking, and more. To address this complexity, NVIDIA has developed an observability AI agent framework leveraging the OODA loop strategy, according to NVIDIA Technical Blog.

AI-Powered Observability Framework

The NVIDIA DGX Cloud team, responsible for a global GPU fleet spanning major cloud service providers and NVIDIA’s own data centers, has implemented this innovative framework. The system enables operators to interact with their data centers, asking questions about GPU cluster reliability and other operational metrics.

For instance, operators can query the system about the top five most frequently replaced parts with supply chain risks or assign technicians to resolve issues in the most vulnerable clusters. This capability is part of a project dubbed LLo11yPop (LLM + Observability), which uses the OODA loop (Observation, Orientation, Decision, Action) to enhance data center management.

Monitoring Accelerated Data Centers

With each new generation of GPUs, the need for comprehensive observability increases. Standard metrics such as utilization, errors, and throughput are just the baseline. To fully understand the operational environment, additional factors like temperature, humidity, power stability, and latency must be considered.

NVIDIA’s system leverages existing observability tools and integrates them with NIM microservices, allowing operators to converse with Elasticsearch in human language. This enables accurate, actionable insights into issues like fan failures across the fleet.

Model Architecture

The framework consists of various agent types:

• Orchestrator agents: Route questions to the appropriate analyst and choose the best action.
• Analyst agents: Convert broad questions into specific queries answered by retrieval agents.
• Action agents: Coordinate responses, such as notifying site reliability engineers (SREs).
• Retrieval agents: Execute queries against data sources or service endpoints.
• Task execution agents: Perform specific tasks, often through workflow engines.

This multi-agent approach mimics organizational hierarchies, with directors coordinating efforts, managers using domain knowledge to allocate work, and workers optimized for specific tasks.

Moving Towards a Multi-LLM Compound Model

To manage the diverse telemetry required for effective cluster management, NVIDIA employs a mixture of agents (MoA) approach. This involves using multiple large language models (LLMs) to handle different types of data, from GPU metrics to orchestration layers like Slurm and Kubernetes.

By chaining together small, focused models, the system can fine-tune specific tasks such as SQL query generation for Elasticsearch, thereby optimizing performance and accuracy.

Autonomous Agents with OODA Loops

The next step involves closing the loop with autonomous supervisor agents that operate within an OODA loop. These agents observe data, orient themselves, decide on actions, and execute them. Initially, human oversight ensures the reliability of these actions, forming a reinforcement learning loop that improves the system over time.

Lessons Learned

Key insights from developing this framework include the importance of prompt engineering over early model training, choosing the right model for specific tasks, and maintaining human oversight until the system proves reliable and safe.

Building Your AI Agent Application

NVIDIA provides various tools and technologies for those interested in building their own AI agents and applications. Resources are available at ai.nvidia.com and detailed guides can be found on the NVIDIA Developer Blog.

Image source: Shutterstock