High Cost & High Risk with AI

Posted on by lechuck park

This image illustrates the high cost and high risk of AI/LLM (Large Language Model) training.

Key Analysis

Left: AI/LLM Growth Path

  • Evolution from Internet → Mobile & Cloud → AI/LLM (Transformer)
  • Each stage shows increasing fluctuations in the graph
  • Emphasizes “High Cost, High Risk” message

Center: Real Problem Visualization

The red graph shows dramatic performance spikes that occurred during actual training processes.

Top Right: Silent Data Corruption (SDC) Issues

Silent data corruption from hardware failures:

  • Power drops, thermal stress → hardware faults
  • Silent errors → training divergence
  • 6 SDC failures in a 54-day pretraining run

Bottom Right: Reliability Issues in Large-Scale ML Clusters (Meta Case)

Real failure cases:

  • 8-GPU job: average 47.7 days
  • 1024-GPU job: MTTF (Mean Time To Failure) 7.9 hours
  • 16,384-GPU job: failure in approximately 1.8 hours

Summary

  1. As GPU scale increases, failure probability rises exponentially, making large-scale AI training extremely costly and technically risky.
  2. Hardware-induced silent data corruption causes training divergence, with 6 failures recorded in just 54 days of pretraining.
  3. Meta’s experience shows massive GPU clusters can fail in under 2 hours, highlighting infrastructure reliability as a critical challenge.

#AITraining #LLM #MachineLearning #DataCorruption #GPUCluster #MLOps #AIInfrastructure #HardwareReliability #TransformerModels #HighPerformanceComputing #AIRisk #MLEngineering #DeepLearning

Big Changes with AI

Posted on by lechuck park

This image illustrates the dramatic growth in computing performance and data throughput from the Internet era to the AI/LLM era.

Key Development Stages

1. Internet Era

  • 10 TWh (terawatt-hours) power consumption
  • 2 PB/day (petabytes/day) data processing
  • 1K DC (1,000 data centers)
  • PUE 3.0 (Power Usage Effectiveness)

2. Mobile & Cloud Era

  • 200 TWh (20x increase)
  • 20,000 PB/day (10,000x increase)
  • 4K DC (4x increase)
  • PUE 1.8 (improved efficiency)

3. AI/LLM (Transformer) Era – “Now Here?” point

  • 400+ TWh (40x additional increase)
  • 1,000,000,000 PB/day = 1 billion PB/day (500,000x increase)
  • 12K DC (12x increase)
  • PUE 1.4 (further improved efficiency)

Summary

The chart demonstrates unprecedented exponential growth in data processing and power consumption driven by AI and Large Language Models. While data center efficiency (PUE) has improved significantly, the sheer scale of computational demands has skyrocketed. This visualization emphasizes the massive infrastructure requirements that modern AI systems necessitate.

#AI #LLM #DataCenter #CloudComputing #MachineLearning #ArtificialIntelligence #BigData #Transformer #DeepLearning #AIInfrastructure #TechTrends #DigitalTransformation #ComputingPower #DataProcessing #EnergyEfficiency

Large Scale Network Driven Design ( Deepseek V3)

Posted on by lechuck park

Deepseek v3 Large-Scale Network Architecture Analysis

This image explains the Multi-Plane Fat-Tree network structure of Deepseek v3.

Core Architecture

1. 8-Plane Architecture

  • Consists of eight independent network channels (highways)
  • Maximizes network bandwidth and distributes traffic for enhanced scalability

2. Fat-Tree Topology

  • Two-layer switch structure:
    • Leaf SW (Leaf Switches): Directly connected to GPUs
    • Spine SW (Spine Switches): Interconnect leaf switches
  • Enables high-speed communication among all nodes (GPUs) while minimizing switch contention

3. GPU/IB NIC Pair

  • Each GPU is paired with a dedicated Network Interface Card (NIC)
  • Each pair is exclusively assigned to one of the eight planes to initiate communication

Communication Methods

NVLink

  • Ultra-high-speed connection between GPUs within the same node
  • Fast data transfer path used for intra-node communication

Cross-plane Traffic

  • Occurs when communication happens between different planes
  • Requires intra-node forwarding through another NIC, PCIe, or NVLink
  • Primary factor that increases latency

Network Optimization Process

The workflow below minimizes latency and prevents network congestion:

  1. Workload Analysis
  2. All to All (analyzing all-to-all communication patterns)
  3. Plane & Layer Set (plane and layer assignment)
  4. Profiling (Hot-path opt K) (hot-path optimization)
  5. Static Routing (Hybrid) (hybrid static routing approach)

Goal: Low latency & no jamming

Scalability

This design is a scale-out network for large-scale distributed training supporting 16,384+ GPUs. Each plane operates independently to maximize overall system throughput.

3-Line Summary

Deepseek v3 uses an 8-plane fat-tree network architecture that connects 16,384+ GPUs through independent communication channels, minimizing contention and maximizing bandwidth. The two-layer switch topology (Spine and Leaf) combined with dedicated GPU-NIC pairs enables efficient traffic distribution across planes. Cross-plane traffic management and hot-path optimization ensure low-latency, high-throughput communication for large-scale AI training.

#DeepseekV3 #FatTreeNetwork #MultiPlane #NetworkArchitecture #ScaleOut #DistributedTraining #AIInfrastructure #GPUCluster #HighPerformanceComputing #NVLink #DataCenterNetworking #LargeScaleAI

With Claude

AI approach

Posted on by lechuck park

Legacy – The Era of Scale-Up

Traditional AI approach showing its limitations:

  • Simple Data: Starting with basic data
  • Simple Data & Logic: Combining data with logic
  • Better Data & Logic: Improving data and logic
  • Complex Data & Logic: Advancing to complex data and logic
  • Near The Limitation: Eventually hitting a fundamental ceiling

This approach gradually increases complexity, but no matter how much it improves, it inevitably runs into fundamental scalability limitations.

AI Works – The Era of Scale-Out

Modern AI transcending the limitations of the legacy approach through a new paradigm:

  • The left side shows the limitations of the old approach
  • The lightbulb icon in the middle represents a paradigm shift (Breakthrough)
  • The large purple box on the right demonstrates a completely different approach:
    • Massive parallel processing of countless “01/10” units (neural network neurons)
    • Horizontal scaling (Scale-Out) instead of sequential complexity increase
    • Fundamentally overcoming the legacy limitations

Key Message

No matter how much you improve the legacy approach, there’s a ceiling. AI breaks through that ceiling with a completely different architecture.

Summary

  • Legacy AI hits fundamental limits by sequentially increasing complexity (Scale-Up)
  • Modern AI uses massive parallel processing architecture to transcend these limitations (Scale-Out)
  • This represents a paradigm shift from incremental improvement to architectural revolution

#AI #MachineLearning #DeepLearning #NeuralNetworks #ScaleOut #Parallelization #AIRevolution #Paradigmshift #LegacyVsModern #AIArchitecture #TechEvolution #ArtificialIntelligence #ScalableAI #DistributedComputing #AIBreakthrough

Optimize LLM

Posted on by lechuck park

LLM Optimization: Integration of Traditional Methods and New Paradigms

Core Message

LLM (Transformer) optimization requires more than just traditional optimization methodologies – new perspectives must be added.

1. Traditional Optimization Methodology (Left Side)

SW (Software) Optimization

  • Data Optimization
    • Structure: Data structure design
    • Copy: Data movement optimization
  • Logics Optimization
    • Algorithm: Efficient algorithm selection
    • Profiling: Performance analysis and bottleneck identification

Characteristics: Deterministic, logical approach

HW (Hardware) Optimization

  • Functions & Speed (B/W): Function and speed/bandwidth optimization
  • Fit For HW: Optimization for existing hardware
  • New HW implementation: New hardware design and implementation

Characteristics: Physical performance improvement focus

2. New Perspectives Required for LLM (Right Side)

SW Aspect: Human-Centric Probabilistic Approach

  • Human Language View / Human’s View
    • Human language understanding methods
    • Human thinking perspective
  • Human Learning
    • Mimicking human learning processes

Key Point: Statistical and Probabilistic Methodology

  • Different from traditional deterministic optimization
  • Language patterns, probability distributions, and context understanding are crucial

HW Aspect: Massive Parallel Processing

  • Massive Simple Parallel
    • Parallel processing of large-scale simple computations
    • Hardware architecture capable of parallel processing (GPU/TPU) is essential

Key Point: Efficient parallel processing of large-scale matrix operations

3. Integrated Perspective

LLM Optimization = Traditional Optimization + New Paradigm

DomainTraditional MethodLLM Additional Elements
SWAlgorithm, data structure optimization+ Probabilistic/statistical approach (human language/learning perspective)
HWFunction/speed optimization+ Massive parallel processing architecture

Conclusion

For effective LLM optimization:

  1. Traditional optimization techniques (data, algorithms, hardware) as foundation
  2. Probabilistic approach reflecting human language and learning methods
  3. Hardware perspective supporting massive parallel processing

These three elements must be organically combined – this is the core message of the diagram.

Summary

LLM optimization requires integrating traditional deterministic SW/HW optimization with new paradigms: probabilistic/statistical approaches that mirror human language understanding and learning, plus hardware architectures designed for massive parallel processing. This represents a fundamental shift from conventional optimization, where human-centric probabilistic thinking and large-scale parallelism are not optional but essential dimensions.

#LLMOptimization #TransformerArchitecture #MachineLearningOptimization #ParallelProcessing #ProbabilisticAI #HumanLanguageView #GPUComputing #DeepLearningHardware #StatisticalML #AIInfrastructure #ModelOptimization #ScalableAI #NeuralNetworkOptimization #AIPerformance #ComputationalEfficiency