Tag: AIOps

AI agent fro DC Operations

Posted on by lechuck park

Data & Knowledge Driven AI Agent for Data Center Operations

This illustration describes how data center operations can evolve from facility data → operational knowledge → AI Agent → automated operations.

The key message is not simply that AI controls data center equipment.

Rather, it shows how AI Agents can connect operational data with accumulated human knowledge, understand operational situations, reason about incidents, and support or automate operational actions.

1. Facilities & Systems

The process starts with the physical infrastructure and operational systems of the data center.

The illustration represents:

  • Power
  • Cooling
  • Network
  • IT / GPU
  • Security
  • Environment

Sensors and systems continuously generate operational information.

Systems such as DCIM, NMS, BMS, and log platforms collect this information.

In simple terms:

Facilities generate information, and systems collect it.

2. Data — From Information to Metrics

Facility information is transformed into measurable operational data.

For example:

  • Temperature → 42.3°C
  • Power Load → 12.6 MW
  • Water Flow → 3.2 m³/h
  • Utilization → 78%

The important point is that the AI Agent uses both:

Real-time Data + Historical Data

Real-time data tells the Agent what is happening now, while historical data provides the operational context and previous experience.

3. Event — Turning Numbers into Meaning

Raw numbers are not always meaningful to operators.

Therefore, data is transformed into understandable events.

For example:

GPU Inlet Temperature is high (42.3°C)

Now the numerical value has become a meaningful operational event.

The event also contains context such as:

What happened + Where + When + Severity + Impact

This is important because the AI Agent does not need to operate only on raw numbers. It can reason about meaningful operational situations.

4. Response — Operational Knowledge

When an event occurs, traditional operations rely on manuals and experienced operators.

The illustration represents this knowledge through:

  • Runbook
  • MOP
  • EOP
  • SOP
  • Best Practices

A typical response process can be:

Check → Analyze → Execute → Verify

This represents the transformation of human experience into reusable operational knowledge.

Human Experience → Documentation → Operational Knowledge

This knowledge becomes one of the most important assets for the AI Agent.

5. Final Decision — Judgment & Action

The final stage goes beyond detecting an event.

The operational process becomes:

Root Cause → Action Plan → Service Restore → Record

Traditionally, experienced operators perform much of this reasoning manually.

With an AI Agent, operational data and knowledge can be combined to support:

  • Root-cause analysis
  • Action recommendations
  • Runbook execution
  • Operator guidance
  • Controlled automation

In a real data center, however, autonomous action should be governed by policies, safety controls, and human approval where required.

The Center of the Illustration — AI Agent

The AI Agent sits at the center because it connects Data and Knowledge.

Data

  • Operational Data
  • Facility & Asset Data
  • Event & Incident Data
  • Historical Cases
  • Asset / Relationship Data

Knowledge

  • Manuals
  • Runbooks
  • SOP / MOP / EOP
  • Domain Knowledge
  • Best Practices
  • Past Cases & Lessons

The Agent combines these two layers to perform:

Learn → Reason → Act

A simple way to express the concept is:

Data tells the Agent what is happening.
Knowledge tells the Agent what it means and what to do.

The Core Message

The most important point of the illustration is that the AI Agent itself is not the foundation.

The real foundation is:

Data → Knowledge → AI Agent → Operations

Without accurate data, the Agent cannot reliably understand the current state.

Without structured operational knowledge, the Agent cannot reliably determine what the situation means or what response is appropriate.

Therefore, the real objective of AI-enabled data center operations is not simply:

“Deploy AI.”

It is:

“Make operational data and knowledge usable by AI.”

The Transformation of Data Center Operations

The bottom of the illustration shows:

Data-Driven → Knowledge-Centric → AI-Powered

This represents the evolution of operational models.

Traditional Operations

Human → Data → Manual Analysis → Manual Action

AI Agent-Based Operations

Data + Knowledge → AI Agent → Reasoning → Recommended / Controlled Action

The role of people does not disappear.

Instead, it changes.

AI handles repetitive monitoring, analysis, and operational assistance, while people focus more on judgment, decision-making, exception handling, and continuous improvement.

This is why the final concept is:

People + AI

rather than simply AI replaces People.

One-Sentence Summary

By connecting data generated from data center facilities with operational knowledge, an AI Agent can understand, reason, and support or automate operational actions—transforming human-centered operations into data- and knowledge-driven intelligent operations.

#AIDC #DataCenter #AIDataCenter #AIAgent #DataDriven #KnowledgeDriven #AIOps #DCIM #DataCenterOperations #OperationalAutomation #DigitalTransformation #DataAndKnowledge #IntelligentOperations #HumanAndAI

With ChatGPT

AI DC Operation Strategy

Posted on by lechuck park

AI DC Operation Strategy Infographic Analysis

This image is a highly technical, professional infographic utilizing a dark blue and teal neon cybernetic aesthetic to illustrate the organic and automated ecosystem of AI Data Center operations. The overall layout features a continuous feedback loop where four core elements dynamically interact around a central command hub.

1. Standardized Data Collection (Top Left) This section represents the entry point for rapidly and accurately ingesting vast amounts of facility data. Depicting server racks alongside technical nodes like Kafka data pipelines, API Gateways, and standardization protocols (e.g., ETSI), it visually demonstrates the high-speed collection and standardization of real-time telemetry data from diverse infrastructure components.

2. AI Agent Automation (Right) Acting as the “brain” of the operation, this area processes the ingested data to execute intelligent control. Centered around a glowing AI brain icon, it highlights machine learning algorithms, reinforcement learning modules, and decision logic engines. It illustrates a system that goes beyond simple monitoring—where AI models continuously learn, run through model optimization cycles, and elevate the level of operational automation.

3. Advanced Power & Pre-emptive Cooling (Bottom Left) This section addresses the physical infrastructure required to handle the high-density heat and power demands typical of AI workloads. It features smart sensor arrays, power distribution units, and thermal management systems. Notably, a chart comparing “Predicted Temp vs. Actual Temp” visually proves the application of pre-emptive cooling algorithms—shifting from reactive cooling to data-driven, proactive thermal mitigation.

4. Integrated Workforce Management (Center) Positioned at the heart of the graphic, this is the Control Hub that orchestrates the entire advanced ecosystem. A team of professionals is shown surrounded by large dashboard monitors. This emphasizes that AI does not replace humans; rather, it empowers a highly skilled workforce to focus on “Human-in-the-Loop Supervision,” strategic analytics, and continuous data-driven improvement across the expanded data center footprint.

📌 Summary

This infographic illustrates that AI Data Center operations have evolved into an “intelligent, autonomous ecosystem.” It showcases a perfect virtuous cycle: Standardized data (1) feeds AI agents (2), which in turn drive proactive power and pre-emptive cooling infrastructure (3). Ultimately, an empowered, highly-skilled workforce (4) strategically orchestrates, verifies, and optimizes this entire continuous loop from a central control hub.

#AIDataCenter #AIOps #DCOperations #InfraAutomation #PreemptiveCooling #Telemetry #IntegratedWorkforce

With Gemini

Metric Changes : Raw to Intelligent

Posted on by lechuck park

This image, titled “Metric Changes : Raw to Intelligent,” illustrates the evolution of IT system monitoring and data analysis across four progressive stages. Moving from left to right, it demonstrates how systems transition from basic, reactive alert mechanisms to smart, predictive operations.

Stage-by-Stage Description

  • Stage 1: Raw Metric
    • Concept: This is the most fundamental monitoring method. It relies on a static, fixed threshold (e.g., Alert: >80%). The primary focus is on basic visibility regarding current status and defects.
    • Goal: Defect Detection
    • Example: If the current CPU usage hits a fixed value of 95%, the system immediately flags it as a “System Bottleneck!!”
  • Stage 2: Delta Metric
    • Concept: Moving beyond static numbers, this stage monitors the rate of change (Delta). It tracks how rapidly a metric fluctuates over a specific timeframe (e.g., Delta >50/min) to catch sudden spikes.
    • Goal: Early Spike Detection
    • Example: If error logs experience a sudden spike of +100 per minute, the system recognizes this rapid change and triggers an “Anomaly Detected!” alert.
  • Stage 3: Trend Metric
    • Concept: This stage utilizes historical data to forecast the future. Instead of a hard number, the threshold becomes “Time-to-Failure.” It calculates the trajectory to determine the exact point of resource exhaustion (T-Exhaustion).
    • Goal: Proactive Response
    • Example: By observing that a disk is filling up at a rate of +2GB/Hour (Time-to-Failure), the system proactively warns that a “Failure < 3H” (failure in less than 3 hours) is imminent.
  • Stage 4: AI Metric
    • Concept: The most advanced stage, utilizing Machine Learning (ML) and Artificial Intelligence. It establishes dynamic thresholds by learning what a “normal” baseline looks like, enabling it to detect complex anomalies and deviations from standard business metrics.
    • Goal: Intelligence & Prediction
    • Example: If a metric exhibits 3X Faster Growth—which acts as a Dynamic Deviation from its learned normal state—the AI intelligently diagnoses it as a “Pattern anomaly!”

📝 Summary

This infographic perfectly visualizes the roadmap of monitoring systems. It highlights the paradigm shift from merely reacting to fixed thresholds, to understanding rates of change and future trends, and ultimately utilizing AI for dynamic, autonomous prediction and intelligent anomaly detection.

#DataAnalysis #SystemMonitoring #AIOps #ArtificialIntelligence #MachineLearning #AnomalyDetection #TrendAnalysis #ITInfrastructure

With Gemini

Ontology+Telemetry

Posted on by lechuck park

The image is titled “Ontology + Telemetry” at the top and is divided into two main columns: a blue-themed section for “Ontology” on the left, and a purple-themed section for “Telemetry” on the right.

1. Left Section: Ontology At the top left, there is an icon of a network graph with connected nodes. The primary focus of this section is “Traceability & Rollback,” which involves the versioning of configuration and history. It details three key components:

  • Graph Versioning (Time-Series Knowledge Graph): Associated with snapshots, event logging, and point-in-time queries. Its main function is “Point-in-time state reconstruction.”
  • IaC (Infrastructure as Code) & GitOps: Focuses on declarative modeling, approval pipelines, and audit trails to enable “Code-driven change approval and audit.”
  • Validation Rules (Integrity Checks): Utilizes schema constraints and auto-filtering to prevent human error, leading to “Automated physical/logical constraint enforcement.”

2. Right Section: Telemetry At the top right, there is an icon depicting line graphs and fluctuating data waves. The primary focus here is “Meaningful Extraction & Data Compression,” managing the lifecycle of trends and anomalies. It also lists three key components:

  • Baseline Management: Uses AIOps and machine learning for contextual normalcy, establishing “ML-driven dynamic thresholds.”
  • Drift Detection: Involves monitoring gradual degradation and enables “Tracking gradual degradation for predictive maintenance.”
  • Data Lifecycle & Roll-up: Deals with downsampling, resolution adjustment, and storage optimization through “Time-based data downsampling.”

💡 Summary
This infographic outlines a comprehensive framework for managing modern IT infrastructure and data centers. It contrasts and combines two essential pillars: “Ontology,” which handles the static configuration, tracing structural changes and rollbacks, and “Telemetry,” which processes dynamic operational metrics to extract meaningful trends and predict anomalies.

#Ontology #Telemetry #ITInfrastructure #DataCenterManagement #AIOps #ConfigurationManagement #PredictiveMaintenance #GitOps

RMC (Rack Management Controller) More

Posted on by lechuck park

This infographic, titled “RMC (Rack Management Controller) More,” details the three advanced core roles and operational capabilities of the RMC (or RMU) in high-density AI data center environments across three color-coded horizontal rows.

1. Power Distribution & Real-Time Telemetry Aggregation

The top pink row covers IT-domain centralized power management and data aggregation.

  • Central Power Shelf Monitoring: Replaces per-server PSUs with a centralized Power Shelf, serving as the physical aggregation point for input/output power telemetry.
  • Data Collection Hub (Aggregator): Aggregates power and thermal data from individual server BMCs and streams metrics to DCIM/BMS via Redfish, IPMI, and SNMP.
  • Proactive Power Prediction: Exposes near-term load forecasting (Power Prediction, 5–10 minutes ahead) to enable preemptive cooling synchronization and load optimization.

2. Liquid Cooling Leak Detection & Automated Safeguards (Safety & Intervention)

The middle green row outlines safety workflows and physical intervention capabilities for liquid cooling architectures.

  • Rack-Level Automated Reaction: Triggers immediate safety workflows upon detecting alerts from rope sensors or manifold leak detection strips.
  • Electrical De-energization: Executes rapid high-voltage isolation before fluid reaches active circuits, coordinating with BMCs to physically cut power at the rack level and prevent short-circuit damage.

3. High-Voltage Power Building Block Orchestration (e.g., Diablo 400 Sidecar)

The bottom blue row highlights hardware-level orchestration across high-voltage power components.

  • BBU & CBU Dynamic Control: Orchestrates battery and capacitor backup modules for grid outage mitigation and instantaneous Peak Shaving during pulse loads.
  • DCPDU Remote Monitoring: Manages per-channel output On/Off switching, Current Limiting, and Ground Fault Detection via standardized RMU interfaces.
  • AC/DC PSU Shelf Coordination: Regulates dynamic power distribution and active feedback control to compensate for busbar voltage drop across high-density AI clusters.

Summary

This infographic highlights the evolution of the RMC from a passive monitoring unit to an active, rack-scale brain. It operates as an IT telemetry neural hub aggregating real-time BMC data and power forecasts, a safety intervention authority enforcing electrical de-energization during liquid cooling leaks, and a power orchestrator managing complex building blocks (BBU, CBU, DCPDU, PSU) in next-generation high-voltage architectures like Diablo 400.

#OCP #OpenRack #ORv3 #RMC #RMU #DataCenterInfrastructure #LiquidCooling #LeakDetection #PeakShaving #Diablo400 #PowerManagement #AIOps #Redfish

With Gemini

GPU Works Monitoring

Posted on by lechuck park

1. The Physical Infrastructure Defense Line (BMC / Out-of-Band)

This is the foundational layer that preemptively monitors the physical environmental limits at the chassis level through a microcontroller (BMC), operating completely independently of the OS or kernel state.

  • Technical Significance: High-density GPU systems are highly sensitive to power spikes and cooling degradation. Before the OS triggers GPU throttling to protect the hardware, this layer must catch anomalies like high-voltage distribution fluctuations or rising return temperatures in liquid/air cooling systems via the System Event Log (SEL).
  • Fault Isolation: It narrows down the root cause by isolating purely physical infrastructure factors—such as “insufficient power supply” or “thermal limits”—before any software-level performance analysis begins.

2. The Hardware Integrity Layer (GPU / In-Band)

This layer tracks the physical aging and data corruption of the High Bandwidth Memory (HBM) and compute cores directly at the chip level, utilizing tools like DCGM (Data Center GPU Manager).

  • Technical Significance: While Single Bit Errors (SBE) within the HBM are auto-correctable, their accumulation strongly indicates memory component aging. Conversely, uncorrectable Double Bit Errors (DBE) or Row Remapping failures due to depleted spare memory banks signify an immediate, fatal interruption to the workload.
  • Fault Isolation: These metrics serve as definitive evidence to immediately isolate (cordon/drain) the affected node from the training cluster and initiate a Return Merchandise Authorization (RMA) with the hardware vendor.

3. The System Logic & Driver Layer (OS/Kernel / In-Band)

This is the logical debugging domain that analyzes the communication state between the NVIDIA device drivers and the Linux kernel, primarily tracking dmesg and XID error logs.

  • Technical Significance: It is crucial to clearly distinguish between software-level crashes caused by user applications (e.g., memory leaks, infinite loops, segfaults) and physical communication disconnections where the GPU stops responding and drops off the PCIe bus (Device Drop-off).
  • Fault Isolation: By separating pure user workload bugs from actual physical device communication failures, this layer eliminates time wasted on unnecessary hardware replacements or node reboots.

4. The Interconnect & Fabric Layer (Interconnect / In-Band)

In a scale-out environment extending beyond a single node, this layer monitors the high-speed data highway for communication bottlenecks.

  • Technical Significance: During large-scale distributed training, a single poor PCIe slot connection or an NVLink CRC integrity check failure can drastically plummet the bandwidth of the entire ring topology. These issues do not crash the system or spit out fatal errors, making them the primary culprits of “Silent Performance Degradation.”
  • Fault Isolation: By tracking PCIe Replay and NVLink Recovery counts in real-time, it pinpoints the exact faulty cables, switch ports, or riser cards causing excessive packet retransmissions among thousands of connections.

Architectural Conclusion

Ultimately, when faced with the single symptom of “a specific node’s computation has slowed down,” you can only pinpoint the true root cause by cross-analyzing Redfish API-based Out-of-Band telemetry with DCGM/dmesg-based In-Band telemetry in real-time.

Moving beyond simple monitoring dashboards, integrating these complex telemetry data streams into an LLM and RAG-based automated agent will serve as a powerful tool to drastically reduce MTTR without requiring manual administrator intervention.

#AIDataCenter #GPUCluster #Telemetry #RootCauseAnalysis #BMC #NVIDIA #DCGM #NVLink #AIOps #InfrastructureAsCode #DataCenterManagement

With Gemini

Co-Work

Posted on by lechuck park

This image, titled “Co-Work,” illustrates a strategic framework for Event-Centric AIOps. It demonstrates how raw telemetry from physical infrastructure is transformed into structured, actionable intelligence for an AI Agent, fundamentally driven by human expertise.

1. Data Generation and Extraction

  • Device to Metric: Physical infrastructure (Device) generates raw operational data.
  • The Role of Configurations: This data is extracted into quantitative Metric (Number) formats. This extraction is guided by Configurations & Topology, which represents the structural configurations and network topology. This ensures the system understands the physical and logical layout of the devices.

2. Contextualization

  • Metric to Context: Raw numerical data lacks operational meaning on its own. It is transformed into readable Context (text), effectively converting raw telemetry into event logs suitable for LLM-based analysis.
  • The Role of System: This conversion is executed by the System, which acts as the Data Processing Operating System. It defines the rules and logic for how raw numbers are processed, correlated, and translated into meaningful operational states.

3. AI Agent Integration

  • Context to AI Agent: The structured, contextualized text is delivered to the AI Agent for analysis, root cause identification, or predictive tasks.
  • The Role of Manual: The AI Agent’s understanding is heavily enriched by the Manual, which encompasses text-based operating manuals, standard operating procedures (SOPs), and historical troubleshooting data. This provides the AI with established guidelines for how to interpret and react to specific scenarios.

4. The Foundation: Human Intent

The green foundational layer, Human Intent, is the most critical aspect of this architecture. Configurations, System, and Manual are the three core elements and systems that are actively built and managed by humans. They dictate the rules, structural layout, and historical knowledge that guide the AI. This ensures that the AI Agent does not operate in a vacuum, but rather functions safely and effectively within the strict boundaries of human operational intent.

Summary

The “Co-Work” architecture visualizes a collaborative AIOps framework where raw device metrics are systematically transformed into contextualized text. By leveraging three key human-managed components—Configurations (topology), Systems (data processing), and Manuals (historical/procedural text)—the architecture bridges the gap between physical hardware and AI. It ensures the AI Agent receives highly structured, context-rich event data to perform accurate and reliable infrastructure management.

#AIOps #EventCentricAIOps #AIDataCenter #HumanInTheLoop #Telemetry #LLM #ITOperations