New Risk @ AI DC

Posted on by lechuck park

Overview: New Risks at AI Data Centers

The image outlines the infrastructure challenges faced by modern AI Data Centers (AI DC), specifically focusing on the high demands placed on hardware like GPUs. It divides these challenges into two primary categories: Power Risk and Cooling Risk.

The central graphic illustrates that the core AI processing units (Brains/GPUs) are entirely dependent on these two foundational elements.

⚡ Power Risk

This section highlights issues related to power supply and infrastructure (such as Power Diversification, ESS, and 800V HVDC).

  • Power Supply Shortage (GPU Power Throttling): When the facility cannot provide enough power, GPUs slow down to compensate.
    • Impacts: Delays in AI workloads, financial losses due to lost data checkpoints, and the collapse of synchronization across the entire computing cluster.
  • Rapid Power Fluctuations: Sudden spikes or drops in the power supply.
    • Impacts: Voltage sag, electrical resonance in external grids, and reduced lifespan or physical damage to backup power systems like generators and UPS (Uninterruptible Power Supplies).
  • Power Quality Degradation: When the provided electricity is “noisy” or unstable.
    • Impacts: Malfunctions in protective electrical relays, overheating of server Power Supply Units (PSUs), and unexplained network communication errors.

❄️ Cooling Risk

This section focuses on the challenges of managing the massive heat generated by AI workloads, specifically looking at Liquid Cooling and changes in Cooling Distribution Unit (CDU) environments.

  • Cooling Supply Shortage (GPU Thermal Throttling): When the cooling system cannot remove heat fast enough, GPUs slow down to prevent melting.
    • Impacts: Delays in AI workloads, reduced lifespan and increased defects in GPUs, and long-term damage to surrounding server equipment.
  • Leakage Occurrence: Physical leaks in the liquid cooling system.
    • Impacts: Immediate equipment burnout (short circuits), risk of electrical arc flashes and fires, and cascading system shutdowns due to a loss of pressure in the cooling loop.
  • Cooling Water Quality Deterioration: When the liquid used for cooling becomes contaminated or degrades.
    • Impacts: Formation of localized “hot-spots” where cooling fails, a sharp decline in overall cooling efficiency, and mechanical wear and tear on the CDU pumps.

📝 Summary

  1. AI Data Centers face critical new infrastructure risks divided into two main categories: supplying massive amounts of power and managing extreme heat.
  2. Power-related risks (shortages, fluctuations, and poor quality) lead to severe workload delays, cluster synchronization failures, and damage to backup generators.
  3. Cooling-related risks (insufficient cooling, leaks, and poor water quality) cause thermal throttling, severe hardware damage, and potentially catastrophic fires.

#AIDataCenter #DataCenterInfrastructure #GPUPower #LiquidCooling #DataCenterRisk #ThermalThrottling #TechInfrastructure

With Gemini

Data Center Changes

Posted on by lechuck park

The Evolution of Data Centers

This infographic, titled “Data Center Changes,” visually explains how data center requirements are skyrocketing due to the shift from traditional computing to AI-driven workloads.

The chart compares three stages of data centers across two main metrics: Rack Density (how much power a single server rack consumes, shown on the vertical axis) and the overall Total Power Capacity (represented by the size and labels of the circles).

  • Traditional DC (Data Center): In the past, data centers ran at a very low rack density of around 2kW. The total power capacity required for a facility was relatively small, at around 10 MW.
  • Cloud-native DC: As cloud computing took over, the demands increased. Rack densities jumped to about 10kW, and the overall facility size grew to require around 100 MW of power.
  • AI DC: This is where we see a massive leap. Driven by heavy GPU workloads, AI data centers push rack densities beyond 100kW+. The scale of these facilities is enormous, demanding up to 1GW of power. The red starburst shape also highlights a new challenge: “Ultra-high Volatility,” meaning the power draw isn’t stable; it spikes violently depending on what the AI is processing.

The Three Core Challenges (Bottom Panels)

The bottom three panels summarize the key takeaways of transitioning to AI Data Centers:

  1. Scale (Massive Investment): Building a 1GW “Campus-scale” AI data center requires astronomical capital expenditure (CAPEX). To put this into perspective, the chart notes that just 10MW costs roughly 200 billion KRW (South Korean Won). Scaling that to 1GW is a colossal financial undertaking.
  2. Density (The Need for Liquid Cooling): Power density per rack is jumping from 2kW to 100kW—a 50x increase. Traditional air-conditioning cannot cool servers running this hot, meaning the industry must transition to advanced liquid cooling technologies.
  3. Volatility (Unpredictable Demands): Unlike traditional servers that run at a steady hum, AI GPU workloads change in real-time. A sudden surge in computing tasks instantly spikes both the electricity needed to run the GPUs and the cooling power needed to keep them from melting.

Summary

  • Data centers are undergoing a massive transformation from Traditional (10MW) and Cloud (100MW) models to gigantic AI Data Centers requiring up to 1 Gigawatt (1GW) of power.
  • Because AI servers use powerful GPUs, power density per rack is increasing 50-fold (up to 100kW+), forcing a shift from traditional air cooling to advanced liquid cooling.
  • This AI infrastructure requires staggering financial investments (CAPEX) and must be designed to handle extreme, real-time volatility in both power and cooling demands.

#DataCenter #AIDataCenter #LiquidCooling #GPU #CloudComputing #TechTrends #TechInfrastructure #CAPEX

With Gemini

The Architecture for AI-Driven Autonomous

Posted on by lechuck park

This slide effectively illustrates a complete, four-tier architecture required to build a fully autonomous AI system. Let’s walk through the framework from the foundation (data collection) to the top (autonomous execution):

  • L1. Ultra-Precision Sensor Layer (The “Sensory Organ”)This foundational layer is all about high-resolution data capture. Acting as the system’s highly sensitive sensory organs, it meticulously monitors minute physical changes—such as heat, flow, and pressure—right down to the individual chipset level.
  • L2. AI-Ready Data Lake (The “Central Library”)Once the data is captured, it flows into this layer to be consolidated. It breaks down data silos by collecting scattered facility data into one centralized library. It then automatically catalogs this information so that the AI can instantly access, read, and learn from it.
  • L3. Pluggable AI Analysis Layer (The “Brain”)This is where the cognitive processing happens. Acting as the brain of the system, it analyzes the organized data to find optimal solutions. Its “pluggable” nature means you can dynamically swap in the best AI algorithms—like Deep Learning or Reinforcement Learning—just like snapping Lego blocks together to fit the specific situation.
  • L4. Autonomous Control Loop (The “Executive Branch”)Finally, the insights from the brain are turned into action here. This layer operates in real-time (down to the millisecond) to send control signals back to the system. It executes decisions entirely on its own, achieving true autonomous operation with zero human intervention.

Summary

This architecture demonstrates a seamless, end-to-end operational flow: it starts by sensing microscopic hardware changes (L1), structures that raw data for immediate AI consumption (L2), applies dynamic and flexible algorithms to make smart decisions (L3), and ultimately executes those decisions autonomously in real-time (L4). It is a perfect blueprint for achieving a fully uncrewed, intelligent infrastructure.

#AIArchitecture #AutonomousSystems #EdgeComputing #DataLake #AIOps #SmartInfrastructure #MachineLearning #Automation

With Gemini

Current Works

Posted on by lechuck park

The proposed AI DC Intelligent Incident Response Platform upgrades traditional data center monitoring to an “Autonomous Operations” system within a secure, air-gapped on-premise environment. It features a Dual-Path architecture that utilizes lightweight LLMs for real-time automated alerts (Fast Path) and high-performance LLMs with GraphRAG for deep root-cause analysis (Slow Path). By structuring fragmented manuals and comprehensively mapping infrastructure dependencies, this system significantly reduces recovery time (MTTR) and provides a highly scalable, cost-effective solution for hyper-scale AI data centers

With NotebookLM

Prefill & Decode (2)

Posted on by lechuck park

1. Prefill Phase (Input Analysis & Parallel Processing)

  • Processing Method: It processes the user’s lengthy prompts or documents all at once using Parallel Computing.
  • Bottleneck (Compute-bound): Since it needs to process a massive amount of data simultaneously, computational power is the most critical factor. This phase generates the KV (Key-Value) cache which is used in the next step.
  • Requirements: Because it requires large-scale parallel computation, massive throughput and large-capacity memory are essential.
  • Hardware Characteristics: The image describes this as a “One-Hit & Big Size” style, explaining that a GPU-based HBM (High Bandwidth Memory) architecture is highly suitable for handling such large datasets.

2. Decode Phase (Sequential Token Generation)

  • Processing Method: Using the KV cache generated during the Prefill phase, this is a Sequential Computing process that generates the response tokens one by one.
  • Bottleneck (Memory-bound): The computation itself is light, but the system must constantly access the memory (KV cache) to fetch and generate the next word. Therefore, memory access speed (bandwidth) becomes the limiting factor.
  • Requirements: Because it needs to provide small and immediate responses to the user, ultra-low latency and deterministic execution speed are crucial.
  • Hardware Characteristics: Described as a “Small-Hit & Fast Size” style, an LPU (Language Processing Unit)-based SRAM architecture is highly advantageous to minimize latency.

💡Summary

  1. Prefill is a compute-bound phase that processes user input in parallel all at once to create a KV cache, making GPU and HBM architectures highly suitable.
  2. Decode is a memory-bound phase that sequentially generates words one by one by referencing the KV cache, where LPU and SRAM architectures are advantageous for achieving ultra-low latency.
  3. Ultimately, an LLM operates by grasping the context through large-scale computation (Prefill) and then generating responses in real-time through fast memory access (Decode).

#LLM #Prefill #Decode #GPU_HBM #LPU_SRAM #AIArchitecture #ParallelComputing #UltraLowLatencyAI

With Gemini