Network for AI

Posted on by lechuck park

1. Core Philosophy: All for Model Optimization

The primary goal is to create an “Architecture that fits the model’s operating structure.” Unlike traditional general-purpose data centers, AI infrastructure is specialized to handle the massive data throughput and synchronized computations required by LLMs (Large Language Models).

2. Hierarchical Network Design

The architecture is divided into two critical layers to handle different levels of data exchange:

A. Inter-Chip Network (Scale-Up)

This layer focuses on the communication between individual GPUs/Accelerators within a single server or node.

  • Key Goals: Minimize data copying and optimize memory utilization (Shared Memory/Memory Pooling).
  • Technologies: * NVLink / NVSwitch: NVIDIA’s proprietary high-speed interconnect.
  • UALink (Ultra Accelerator Link): The new open standard designed for scale-up AI clusters.

B. Inter-Server Network (Scale-Out)

This layer connects multiple server nodes to form a massive AI cluster.

  • Key Goals: Achieve “No Latency” (Ultra-low latency) and minimize routing overhead to prevent bottlenecks during collective communications (e.g., All-Reduce).
  • Technologies: * InfiniBand: A lossless, high-bandwidth fabric preferred for its low CPU overhead.
  • RoCE (RDMA over Converged Ethernet): High-speed Ethernet that allows direct memory access between servers.

3. Zero Trust Security & Physical Separation

A unique aspect of this architecture is the treatment of security.

  • Operational Isolation: The security and management plane is completely separated from the model operation plane.
  • Performance Integrity: By being physically separated, security protocols (like firewalls or encryption inspection) do not introduce latency into the high-speed compute fabric where the model runs. This ensures that a “Zero Trust” posture does not degrade training or inference speed.

4. Architectural Feedback Loop

The arrow at the bottom indicates a feedback loop: the performance metrics and requirements of the inter-chip and inter-server networks directly inform the ongoing optimization of the overall architecture. This ensures the platform evolves alongside advancing AI model structures.

The architecture prioritizes model-centric optimization, ensuring infrastructure is purpose-built to match the specific operating requirements of large-scale AI workloads.

It employs a dual-tier network strategy using Inter-chip (NVLink/UALink) for memory efficiency and Inter-server (InfiniBand/RoCE) for ultra-low latency cluster scaling.

Zero Trust security is integrated through complete physical separation from the compute fabric, allowing for robust protection without causing any performance bottlenecks.

#AIDC #ArtificialIntelligence #GPU #Networking #NVLink #UALink #InfiniBand #RoCEv2 #ZeroTrust #DataCenterArchitecture #MachineLearningOps #ScaleOut

Redfish for AI DC

Posted on by lechuck park

This image illustrates the pivotal role of the Redfish API (developed by DMTF) as the standardized management backbone for modern AI Data Centers (AI DC). As AI workloads demand unprecedented levels of power and cooling, Redfish moves beyond traditional server management to provide a unified framework for the entire infrastructure stack.

1. Management & Security Framework (Left Column)

  • Unified Multi-Vendor Management:
    • Acts as a single, standardized API to manage diverse hardware from different vendors (NVIDIA, AMD, Intel, etc.).
    • It reduces operational complexity by replacing fragmented, vendor-specific IPMI or OEM extensions with a consistent interface.
  • Modern Security Framework:
    • Designed for multi-tenant AI environments where security is paramount.
    • Supports robust protocols like session-based authentication, X.509 certificates, and RBAC (Role-Based Access Control) to ensure only authorized entities can modify critical infrastructure.
  • Precision Telemetry:
    • Provides high-granularity, real-time data collection for voltage, current, and temperature.
    • This serves as the foundation for energy efficiency optimization and fine-tuning performance based on real-time hardware health.

2. Infrastructure & Hardware Control (Right Column)

  • Compute / Accelerators:
    • Enables per-GPU instance power capping, allowing operators to limit power consumption at a granular level.
    • Monitors the health of high-speed interconnects like NVLink and PCIe switches, and simplifies firmware lifecycle management across the cluster.
  • Liquid Cooling:
    • As AI chips run hotter, Redfish integrates with CDU (Cooling Distribution Unit) systems to monitor pump RPM and loop pressure.
    • It includes critical safety features like leak detection sensors and integrated event handling to prevent hardware damage.
  • Power Infrastructure:
    • Extends management to the rack level, including Smart PDU outlet metering and OCP (Open Compute Project) Power Shelf load balancing.
    • Facilitates advanced efficiency analytics to drive down PUE (Power Usage Effectiveness).

Summary

For an AI DC Optimization Architect, Redfish is the essential “language” that enables Software-Defined Infrastructure. By moving away from manual, siloed hardware management and toward this API-driven approach, data centers can achieve the extreme automation required to shift OPEX structures predominantly toward electricity costs rather than labor.

#AIDataCenter #RedfishAPI #DMTF #DataCenterInfrastructure #GPUComputing #LiquidCooling #SustainableIT #SmartPDU #OCP #InfrastructureAutomation #TechArchitecture #EnergyEfficiency

With Gemini

DC Digitalizations with ISA-95

Posted on by lechuck park

5-Layer Breakdown of DC Digitalization

M1: Sensing & Manipulation (ISA-95 Level 0-1)

  • Focus: Bridging physical assets with digital systems.
  • Key Activities: Ultra-fast data collection and hardware actuation.
  • Examples: High-frequency power telemetry (ms-level), precision liquid cooling control, and PTP (Precision Time Protocol) for synchronization.

M2: Monitoring & Supervision (ISA-95 Level 2)

  • Focus: Holistic visibility and IT/OT Convergence.
  • Key Activities: Correlating physical facility health (cooling/power) with IT workload performance.
  • Examples: Integrated dashboards (“Single Pane of Glass”), GPU telemetry via DCGM, and real-time anomaly detection.

M3: Manufacturing Operations Management (ISA-95 Level 3)

  • Focus: Operational efficiency and workload orchestration.
  • Key Activities: Maximizing “production” (AI output) through intelligent scheduling.
  • Examples: Topology-aware scheduling, AI-OEE (maximizing Model Flops Utilization), and predictive maintenance for assets.

M4: Business Planning & Logistics (ISA-95 Level 4)

  • Focus: Strategic planning, FinOps, and cost management.
  • Key Activities: Managing business logic, forecasting capacity, and financial tracking.
  • Examples: Per-token billing, SLA management with performance guarantees, and ROI analysis on energy procurement.

M5: AI Orchestration & Optimization (Cross-Layer)

  • Focus: Autonomous optimization (AI for AI Ops).
  • Key Activities: Using ML to predictively control infrastructure and bridge the gap between thermal inertia and dynamic loads.
  • Examples: Predictive cooling (cooling down before a heavy job starts), Digital Twins, and Carbon-aware scheduling (ESG).

Summary of Core Concepts

  • IT/OT Convergence: Integrating Information Technology (servers/software) with Operational Technology (power/cooling).
  • AI-OEE: Adapting the “Overall Equipment Effectiveness” metric from manufacturing to measure how efficiently a DC produces AI models.
  • Predictive Control: Moving from reactive monitoring to proactive, AI-driven management of power and heat.

#DataCenter #DigitalTransformation #ISA95 #AIOps #SmartFactory #ITOTConvergence #SustainableIT #GPUOrchestration #FinOps #LiquidCooling

With Gemini

MPFT: Multi-Plane Fat-Tree for Massive Scale and Cost Efficiency

Posted on by lechuck park

MPFT: Multi-Plane Fat-Tree for Massive Scale and Cost Efficiency

1. Architecture Overview (Blue Section)

The core innovation of MPFT lies in parallelizing network traffic across multiple independent “planes” to maximize bandwidth and minimize hardware overhead.

  • Multi-Plane Architecture: The network is split into 4 independent planes (channels).
  • Multiple Physical Ports per NIC: Each Network Interface Card (NIC) is equipped with multiple ports—one for each plane.
  • QP Parallel Utilization (Packet Striping): A single Queue Pair (QP) can utilize all available ports simultaneously. This allows for striped traffic, where data is spread across all paths at once.
  • Out-of-Order Placement: Because packets travel via different planes, they may arrive in a different order than they were sent. Therefore, the NIC must natively support out-of-order processing to reassemble the data correctly.

2. Performance & Cost Results (Purple Section)

The table compares MPFT against standard topologies like FT2/FT3 (Fat-Tree), SF (Slim Fly), and DF (Dragonfly).

MetricMPFTFT3Dragonfly (DF)
Endpoints16,38465,536261,632
Switches7685,12016,352
Total Cost$72M$491M$1,522M
Cost per Endpoint$4.39k$7.5k$5.8k
  • Scalability: MPFT supports 16,384 endpoints, which is significantly higher than a standard 2-tier Fat-Tree (FT2).
  • Resource Efficiency: It achieves high scalability while using far fewer switches (768) and links compared to the 3-tier Fat-Tree (FT3).
  • Economic Advantage: At $4.39k per endpoint, it is one of the most cost-efficient models for large-scale data centers, especially when compared to the $7.5k cost of FT3.

Summary

MPFT is presented as a “sweet spot” solution for AI/HPC clusters. It provides the high-speed performance of complex 3-tier networks but keeps the cost and hardware complexity closer to simpler 2-tier systems by using multi-port NICs and traffic striping.

#NetworkArchitecture #DataCenter #HighPerformanceComputing #GPU #AITraining #MultiPlaneFatTree #MPFT #NetworkingTech #ClusterComputing #CloudInfrastructure