Tag: CloudInfrastructure

Power/Cooling in the linux kernel

Posted on by lechuck park

1. Power Capping Framework

  • Objective (Center): Prevents power grid overload and cuts electricity costs during peak hours.
  • Mechanism (Right): Enforces a strict upper limit on total server power consumption based on DCIM (Datacenter Infrastructure Management) demands.
    1. The DCIM grid signals a heavy load status.
    2. The Linux kernel receives the specific power capping command.
    3. The kernel immediately drops processor clocks and voltages in milliseconds to protect the local power grid.

2. Thermal Subsystem

  • Objective (Center): Prevents hardware overheating and balances the load on external cooling infrastructure, such as Coolant Distribution Units (CDUs) and chillers.
  • Mechanism (Right): Maps temperature-sensing ‘Thermal Zones’ directly to hardware ‘Cooling Devices’ for unified, holistic control.
    1. Hardware sensors detect sudden spikes in internal temperature.
    2. The kernel dynamically adjusts internal server fans and triggers safety throttling.
    3. Temperature telemetry data is actively shared with the external datacenter CDU to ramp up liquid coolant flow rates.

3. Thermal-Aware / Energy-Aware Scheduling

  • Objective (Center): Eliminates physical ‘Hotspots’ within the server room layout and optimizes overall air conditioning (AC) power efficiency.
  • Mechanism (Right): Distributes heavy workloads away from physical servers trapped in low-cooling zones to servers located in cooler zones.
    1. The localized ambient temperature around a specific server rack rises.
    2. The datacenter orchestrator and the kernel work together to throttle the target CPU’s capacity weights.
    3. The Linux scheduler automatically migrates heavy compute tasks to cooler servers across the room in real-time.

Modern Linux has evolved beyond managing isolated servers; it now acts as a holistic orchestrator that treats the datacenter’s power grid, liquid cooling loops, and air conditioning as a single, unified organism.

#LinuxKernel #PowerManagement #ThermalSubsystem #EnergyAwareScheduling #DatacenterInfrastructure #DCIM #LiquidCooling #GreenComputing #HPC #InfrastructureAutomation #CloudInfrastructure

With Gemini

MPFT: Multi-Plane Fat-Tree for Massive Scale and Cost Efficiency

Posted on by lechuck park

MPFT: Multi-Plane Fat-Tree for Massive Scale and Cost Efficiency

1. Architecture Overview (Blue Section)

The core innovation of MPFT lies in parallelizing network traffic across multiple independent “planes” to maximize bandwidth and minimize hardware overhead.

  • Multi-Plane Architecture: The network is split into 4 independent planes (channels).
  • Multiple Physical Ports per NIC: Each Network Interface Card (NIC) is equipped with multiple ports—one for each plane.
  • QP Parallel Utilization (Packet Striping): A single Queue Pair (QP) can utilize all available ports simultaneously. This allows for striped traffic, where data is spread across all paths at once.
  • Out-of-Order Placement: Because packets travel via different planes, they may arrive in a different order than they were sent. Therefore, the NIC must natively support out-of-order processing to reassemble the data correctly.

2. Performance & Cost Results (Purple Section)

The table compares MPFT against standard topologies like FT2/FT3 (Fat-Tree), SF (Slim Fly), and DF (Dragonfly).

MetricMPFTFT3Dragonfly (DF)
Endpoints16,38465,536261,632
Switches7685,12016,352
Total Cost$72M$491M$1,522M
Cost per Endpoint$4.39k$7.5k$5.8k
  • Scalability: MPFT supports 16,384 endpoints, which is significantly higher than a standard 2-tier Fat-Tree (FT2).
  • Resource Efficiency: It achieves high scalability while using far fewer switches (768) and links compared to the 3-tier Fat-Tree (FT3).
  • Economic Advantage: At $4.39k per endpoint, it is one of the most cost-efficient models for large-scale data centers, especially when compared to the $7.5k cost of FT3.

Summary

MPFT is presented as a “sweet spot” solution for AI/HPC clusters. It provides the high-speed performance of complex 3-tier networks but keeps the cost and hardware complexity closer to simpler 2-tier systems by using multi-port NICs and traffic striping.

#NetworkArchitecture #DataCenter #HighPerformanceComputing #GPU #AITraining #MultiPlaneFatTree #MPFT #NetworkingTech #ClusterComputing #CloudInfrastructure