Data Center Changes

Posted on by lechuck park

The Evolution of Data Centers

This infographic, titled “Data Center Changes,” visually explains how data center requirements are skyrocketing due to the shift from traditional computing to AI-driven workloads.

The chart compares three stages of data centers across two main metrics: Rack Density (how much power a single server rack consumes, shown on the vertical axis) and the overall Total Power Capacity (represented by the size and labels of the circles).

  • Traditional DC (Data Center): In the past, data centers ran at a very low rack density of around 2kW. The total power capacity required for a facility was relatively small, at around 10 MW.
  • Cloud-native DC: As cloud computing took over, the demands increased. Rack densities jumped to about 10kW, and the overall facility size grew to require around 100 MW of power.
  • AI DC: This is where we see a massive leap. Driven by heavy GPU workloads, AI data centers push rack densities beyond 100kW+. The scale of these facilities is enormous, demanding up to 1GW of power. The red starburst shape also highlights a new challenge: “Ultra-high Volatility,” meaning the power draw isn’t stable; it spikes violently depending on what the AI is processing.

The Three Core Challenges (Bottom Panels)

The bottom three panels summarize the key takeaways of transitioning to AI Data Centers:

  1. Scale (Massive Investment): Building a 1GW “Campus-scale” AI data center requires astronomical capital expenditure (CAPEX). To put this into perspective, the chart notes that just 10MW costs roughly 200 billion KRW (South Korean Won). Scaling that to 1GW is a colossal financial undertaking.
  2. Density (The Need for Liquid Cooling): Power density per rack is jumping from 2kW to 100kW—a 50x increase. Traditional air-conditioning cannot cool servers running this hot, meaning the industry must transition to advanced liquid cooling technologies.
  3. Volatility (Unpredictable Demands): Unlike traditional servers that run at a steady hum, AI GPU workloads change in real-time. A sudden surge in computing tasks instantly spikes both the electricity needed to run the GPUs and the cooling power needed to keep them from melting.

Summary

  • Data centers are undergoing a massive transformation from Traditional (10MW) and Cloud (100MW) models to gigantic AI Data Centers requiring up to 1 Gigawatt (1GW) of power.
  • Because AI servers use powerful GPUs, power density per rack is increasing 50-fold (up to 100kW+), forcing a shift from traditional air cooling to advanced liquid cooling.
  • This AI infrastructure requires staggering financial investments (CAPEX) and must be designed to handle extreme, real-time volatility in both power and cooling demands.

#DataCenter #AIDataCenter #LiquidCooling #GPU #CloudComputing #TechTrends #TechInfrastructure #CAPEX

With Gemini

The Architecture for AI-Driven Autonomous

Posted on by lechuck park

This slide effectively illustrates a complete, four-tier architecture required to build a fully autonomous AI system. Let’s walk through the framework from the foundation (data collection) to the top (autonomous execution):

  • L1. Ultra-Precision Sensor Layer (The “Sensory Organ”)This foundational layer is all about high-resolution data capture. Acting as the system’s highly sensitive sensory organs, it meticulously monitors minute physical changes—such as heat, flow, and pressure—right down to the individual chipset level.
  • L2. AI-Ready Data Lake (The “Central Library”)Once the data is captured, it flows into this layer to be consolidated. It breaks down data silos by collecting scattered facility data into one centralized library. It then automatically catalogs this information so that the AI can instantly access, read, and learn from it.
  • L3. Pluggable AI Analysis Layer (The “Brain”)This is where the cognitive processing happens. Acting as the brain of the system, it analyzes the organized data to find optimal solutions. Its “pluggable” nature means you can dynamically swap in the best AI algorithms—like Deep Learning or Reinforcement Learning—just like snapping Lego blocks together to fit the specific situation.
  • L4. Autonomous Control Loop (The “Executive Branch”)Finally, the insights from the brain are turned into action here. This layer operates in real-time (down to the millisecond) to send control signals back to the system. It executes decisions entirely on its own, achieving true autonomous operation with zero human intervention.

Summary

This architecture demonstrates a seamless, end-to-end operational flow: it starts by sensing microscopic hardware changes (L1), structures that raw data for immediate AI consumption (L2), applies dynamic and flexible algorithms to make smart decisions (L3), and ultimately executes those decisions autonomously in real-time (L4). It is a perfect blueprint for achieving a fully uncrewed, intelligent infrastructure.

#AIArchitecture #AutonomousSystems #EdgeComputing #DataLake #AIOps #SmartInfrastructure #MachineLearning #Automation

With Gemini

Current Works

Posted on by lechuck park

The proposed AI DC Intelligent Incident Response Platform upgrades traditional data center monitoring to an “Autonomous Operations” system within a secure, air-gapped on-premise environment. It features a Dual-Path architecture that utilizes lightweight LLMs for real-time automated alerts (Fast Path) and high-performance LLMs with GraphRAG for deep root-cause analysis (Slow Path). By structuring fragmented manuals and comprehensively mapping infrastructure dependencies, this system significantly reduces recovery time (MTTR) and provides a highly scalable, cost-effective solution for hyper-scale AI data centers

With NotebookLM

Prefill & Decode (2)

Posted on by lechuck park

1. Prefill Phase (Input Analysis & Parallel Processing)

  • Processing Method: It processes the user’s lengthy prompts or documents all at once using Parallel Computing.
  • Bottleneck (Compute-bound): Since it needs to process a massive amount of data simultaneously, computational power is the most critical factor. This phase generates the KV (Key-Value) cache which is used in the next step.
  • Requirements: Because it requires large-scale parallel computation, massive throughput and large-capacity memory are essential.
  • Hardware Characteristics: The image describes this as a “One-Hit & Big Size” style, explaining that a GPU-based HBM (High Bandwidth Memory) architecture is highly suitable for handling such large datasets.

2. Decode Phase (Sequential Token Generation)

  • Processing Method: Using the KV cache generated during the Prefill phase, this is a Sequential Computing process that generates the response tokens one by one.
  • Bottleneck (Memory-bound): The computation itself is light, but the system must constantly access the memory (KV cache) to fetch and generate the next word. Therefore, memory access speed (bandwidth) becomes the limiting factor.
  • Requirements: Because it needs to provide small and immediate responses to the user, ultra-low latency and deterministic execution speed are crucial.
  • Hardware Characteristics: Described as a “Small-Hit & Fast Size” style, an LPU (Language Processing Unit)-based SRAM architecture is highly advantageous to minimize latency.

💡Summary

  1. Prefill is a compute-bound phase that processes user input in parallel all at once to create a KV cache, making GPU and HBM architectures highly suitable.
  2. Decode is a memory-bound phase that sequentially generates words one by one by referencing the KV cache, where LPU and SRAM architectures are advantageous for achieving ultra-low latency.
  3. Ultimately, an LLM operates by grasping the context through large-scale computation (Prefill) and then generating responses in real-time through fast memory access (Decode).

#LLM #Prefill #Decode #GPU_HBM #LPU_SRAM #AIArchitecture #ParallelComputing #UltraLowLatencyAI

With Gemini

GPU Heat Risk in 60 secs.

Posted on by lechuck park

1. Intuitive Visual Context (Top Section)

  • The left side depicts a scenario where a normal cooling system (fan icon) completely stops functioning, indicated by the red “X” and arrow.
  • This flow visually demonstrates that the GPU chip on the right is immediately exposed to uncontrollable heat (represented by the red heat waves and the red bar at the bottom).
  • The powerful slogan on the right, “Everything is on the line in 60 seconds,” serves as a stark warning that all infrastructure and data are at critical risk if no action is taken within one minute.

2. Four Critical Stages of Damage Over Time (Bottom Table)

The slide neatly structures each stage based on elapsed time and risk level. Highlighting the core damage elements in purple effectively draws the audience’s attention to the most critical impacts.

  • Stage 1 (Approx. 5 to 10 seconds): Service Paralysis due to Thermal Throttling
    • Temperature: Approx. 87°C to 90°C
    • Impact: Due to the rapid temperature spike, the GPU automatically throttles its performance, causing immediate service paralysis.
  • Stage 2 (Approx. 10 to 20 seconds): Forced Shutdown & Data Evaporation
    • Temperature: Approx. 100°C to 105°C (Hardware thermal limit)
    • Impact: Power is forcibly cut to protect the hardware, resulting in the permanent evaporation of unsaved checkpoint data.
  • Stage 3 (Approx. 30 to 60 seconds onwards): Lifespan Reduction and Failure of Surrounding Components
    • Temperature: Server internal ambient 60°C ~ 80°C+ / Surrounding components 95°C ~ 110°C+
    • Impact: Even after the shutdown, massive residual heat trapped in the rack acts like an oven, severely reducing the lifespan of surrounding components (memory, cables, VRMs) and causing future failures.
  • Stage 4 (Approx. 20 to 30 seconds onwards): Physical Damage (Extreme Scenario)
    • Temperature: 120°C to 150°C or higher
    • Impact: In the worst-case scenario where protection circuits fail, the GPU chip and board physically burn out, leading to permanent hardware destruction.

This slide clearly demonstrates that human intervention is simply too slow to stop thermal runaway in high-density environments. It provides a compelling justification for deploying an intelligent, automated solution that can monitor systems second-by-second and execute immediate emergency protocols.

  1. A total cooling failure in a high-density GPU environment leads to critical service throttling and data loss within a mere 10 to 20 seconds.
  2. Even after a forced shutdown, trapped residual heat continues to severely damage surrounding components, drastically reducing infrastructure lifespan.
  3. This extremely narrow 60-second window proves that human intervention is impossible, making an automated, immediate emergency response agent absolutely essential.

#DataCenter #GPU #ThermalRunaway #CoolingFailure #AIDA #DataCenterAutomation #ITInfrastructure #DisasterRecovery

With Gemini