Tag: MachineLearning

vLLM Features

Posted on by lechuck park

vLLM Features & Architecture Breakdown

This chart outlines the key components of vLLM (Virtual Large Language Model), a library designed to optimize the inference speed and memory efficiency of Large Language Models (LLMs).

1. Core Algorithm

  • PagedAttention
    • Concept: Applies the operating system’s (OS) virtual memory paging mechanism to the attention mechanism.
    • Benefit: It resolves memory fragmentation and enables the storage of the KV (Key-Value) cache in non-contiguous memory spaces, significantly reducing memory waste.

2. Data Unit

  • Block (Page)
    • Concept: The minimum KV cache unit with a fixed token size (e.g., 16 tokens).
    • Benefit: Increases management efficiency via fixed-size allocation and minimizes wasted space (internal fragmentation) within slots.
  • Block Table
    • Concept: A mapping table that connects Logical Blocks to Physical Blocks.
    • Benefit: Allows non-contiguous physical memory to be processed as if it were a continuous context.

3. Operation

  • Pre-allocation (Profiling)
    • Concept: Reserves the maximum required VRAM at startup by running a dummy simulation.
    • Benefit: Eliminates the overhead of runtime memory allocation/deallocation and prevents Out Of Memory (OOM) errors at the source.

4. Memory Handling

  • Swapping
    • Concept: Offloads data to CPU RAM when GPU memory becomes full.
    • Benefit: Handles traffic bursts without server downtime and preserves the context of suspended (waiting) requests.
  • Recomputation
    • Concept: Recalculates data instead of swapping it when recalculation is more cost-effective.
    • Benefit: Optimizes performance for short prompts or in environments with slow interconnects (e.g., PCIe limits).

5. Scheduling

  • Continuous Batching
    • Concept: Iteration-level scheduling that fills idle slots immediately without waiting for other requests to finish.
    • Benefit: Eliminates GPU idle time and maximizes overall throughput.

Summary

  1. vLLM adapts OS memory management techniques (like Paging and Swapping) to optimize LLM serving, solving critical memory fragmentation issues.
  2. Key technologies like PagedAttention and Continuous Batching minimize memory waste and eliminate GPU idle time to maximize throughput.
  3. This architecture ensures high performance and stability by preventing memory crashes (OOM) and efficiently handling traffic bursts.

#vLLM #LLMInference #PagedAttention #AIArchitecture #GPUOptimization #MachineLearning #SystemDesign #AIInfrastructure

With Gemini

Parallelism (1) – Data , Expert

Posted on by lechuck park

Parallelism Comparison: Data Parallelism vs Expert Parallelism

This image compares two major parallelization strategies used for training large language models (LLMs).

Left: Data Parallelism

Structure:

  • Data is divided into multiple batches from the database
  • Same complete model is replicated on each GPU
  • Each GPU independently processes different data batches
  • Results are aggregated to generate final output

Characteristics:

  • Scaling axis: Number of batches/samples
  • Pattern: Full model copy on each GPU, dense training
  • Communication: Gradient All-Reduce synchronization once per step
  • Advantages: Simple and intuitive implementation
  • Disadvantages: Model size must fit in single GPU memory

Right: Expert Parallelism

Structure:

  • Data is divided by layers
  • Tokens are distributed to appropriate experts through All-to-All network and router
  • Different expert models (A, B, C) are placed on each GPU
  • Parallel processing at block/thread level in GPU pool

Characteristics:

  • Scaling axis: Number of experts
  • Pattern: Sparse structure – only few experts activated per token
  • Goal: Maintain large capacity while limiting FLOPs per token
  • Communication: All-to-All token routing
  • Advantages: Can scale model capacity significantly (MoE – Mixture of Experts architecture)
  • Disadvantages: High communication overhead and complex load balancing

Key Differences

AspectData ParallelismExpert Parallelism
Model DivisionFull model replicationModel divided into experts
Data DivisionBatch-wiseLayer/token-wise
Communication PatternGradient All-ReduceToken All-to-All
ScalabilityProportional to data sizeProportional to expert count
EfficiencyDense computationSparse computation (conditional activation)

These two approaches are often used together in practice, enabling ultra-large-scale model training through hybrid parallelization strategies.

Summary

Data Parallelism replicates the entire model across GPUs and divides the training data, synchronizing gradients after each step – simple but memory-limited. Expert Parallelism divides the model into specialized experts and routes tokens dynamically, enabling massive scale through sparse activation. Modern systems combine both strategies to train trillion-parameter models efficiently.

#MachineLearning #DeepLearning #LLM #Parallelism #DistributedTraining #DataParallelism #ExpertParallelism #MixtureOfExperts #MoE #GPU #ModelTraining #AIInfrastructure #ScalableAI #NeuralNetworks #HPC

Linux kernel for GPU Workload

Posted on by lechuck park

Linux Kernel GPU Workload Support Features

Goal: Maximize Memory Efficiency & Data Transfer

The core objective is to treat GPUs as a top-tier component like CPUs, reducing memory bottlenecks for large-scale AI workloads.

Key Features

1. Full CXL (Compute Express Link) Support

  • Standard interface for high-speed connections between CPUs, accelerators (GPU, FPGA), and memory expansion devices
  • Enables high-speed data transfer

2. Enhanced HMM (Heterogeneous Memory Management)

  • Heterogeneous memory management capabilities
  • Allows device drivers to map system memory pages to GPU page tables
  • Enables seamless GPU memory access

3. Enhanced P2P DMA & GPUDirect Support

  • Enables direct data exchange between GPUs
  • Direct communication with NVMe storage and network cards (GPUDirect RDMA)
  • Operates without CPU intervention for improved performance

4. DRM Scheduler & GPU Driver Improvements

  • Enhanced Direct Rendering Manager scheduling functionality
  • Active integration of latest drivers from major vendors: AMD (AMDGPU), Intel (i915/Xe), Intel Gaudi/Ponte Vecchio
  • NVIDIA still uses proprietary drivers

5. Advanced Async I/O via io_uring

  • Efficient I/O request exchange with kernel through Ring Buffer mechanism
  • Optimized asynchronous I/O performance

Summary

The Linux kernel now enables GPUs to independently access memory (CXL, HMM), storage, and network resources (P2P DMA, GPUDirect) without CPU involvement. Enhanced drivers from AMD, Intel, and improved schedulers optimize GPU workload management. These features collectively eliminate CPU bottlenecks, making the kernel highly efficient for large-scale AI and HPC workloads.

#LinuxKernel #GPU #AI #HPC #CXL #HMM #GPUDirect #P2PDMA #AMDGPU #IntelGPU #MachineLearning #HighPerformanceComputing #DRM #io_uring #HeterogeneousComputing #DataCenter #CloudComputing

With Claude

Big Changes with AI

Posted on by lechuck park

This image illustrates the dramatic growth in computing performance and data throughput from the Internet era to the AI/LLM era.

Key Development Stages

1. Internet Era

  • 10 TWh (terawatt-hours) power consumption
  • 2 PB/day (petabytes/day) data processing
  • 1K DC (1,000 data centers)
  • PUE 3.0 (Power Usage Effectiveness)

2. Mobile & Cloud Era

  • 200 TWh (20x increase)
  • 20,000 PB/day (10,000x increase)
  • 4K DC (4x increase)
  • PUE 1.8 (improved efficiency)

3. AI/LLM (Transformer) Era – “Now Here?” point

  • 400+ TWh (40x additional increase)
  • 1,000,000,000 PB/day = 1 billion PB/day (500,000x increase)
  • 12K DC (12x increase)
  • PUE 1.4 (further improved efficiency)

Summary

The chart demonstrates unprecedented exponential growth in data processing and power consumption driven by AI and Large Language Models. While data center efficiency (PUE) has improved significantly, the sheer scale of computational demands has skyrocketed. This visualization emphasizes the massive infrastructure requirements that modern AI systems necessitate.

#AI #LLM #DataCenter #CloudComputing #MachineLearning #ArtificialIntelligence #BigData #Transformer #DeepLearning #AIInfrastructure #TechTrends #DigitalTransformation #ComputingPower #DataProcessing #EnergyEfficiency

AI approach

Posted on by lechuck park

Legacy – The Era of Scale-Up

Traditional AI approach showing its limitations:

  • Simple Data: Starting with basic data
  • Simple Data & Logic: Combining data with logic
  • Better Data & Logic: Improving data and logic
  • Complex Data & Logic: Advancing to complex data and logic
  • Near The Limitation: Eventually hitting a fundamental ceiling

This approach gradually increases complexity, but no matter how much it improves, it inevitably runs into fundamental scalability limitations.

AI Works – The Era of Scale-Out

Modern AI transcending the limitations of the legacy approach through a new paradigm:

  • The left side shows the limitations of the old approach
  • The lightbulb icon in the middle represents a paradigm shift (Breakthrough)
  • The large purple box on the right demonstrates a completely different approach:
    • Massive parallel processing of countless “01/10” units (neural network neurons)
    • Horizontal scaling (Scale-Out) instead of sequential complexity increase
    • Fundamentally overcoming the legacy limitations

Key Message

No matter how much you improve the legacy approach, there’s a ceiling. AI breaks through that ceiling with a completely different architecture.

Summary

  • Legacy AI hits fundamental limits by sequentially increasing complexity (Scale-Up)
  • Modern AI uses massive parallel processing architecture to transcend these limitations (Scale-Out)
  • This represents a paradigm shift from incremental improvement to architectural revolution

#AI #MachineLearning #DeepLearning #NeuralNetworks #ScaleOut #Parallelization #AIRevolution #Paradigmshift #LegacyVsModern #AIArchitecture #TechEvolution #ArtificialIntelligence #ScalableAI #DistributedComputing #AIBreakthrough

From Tokenization to Output

Posted on by lechuck park

From Tokenization to Output: Understanding NLP and Transformer Models

This image illustrates the complete process from tokenization to output in Natural Language Processing (NLP) and transformer models.

Top Section: Traditional Information Retrieval Process (Green Boxes)

  1. Distinction (Difference) – Clear Boundary
    • Cutting word pieces, attaching number tags, creating manageable units, generating receipt slips
  2. Classification (Similarity)
    • Placing in the same neighborhood, gathering similar meanings, classifying by topic on bookshelves, organizing by close proximity
  3. Indexing
    • Remembering position, assigning bookshelf numbers, creating a table of contents, organizing context
  4. Retrieval (Fetching)
    • Asking a question, searching the table of contents, retrieving content, finding necessary information
  5. Processing → Result
    • Analyzing information, synthesizing content, writing a report, generating the final answer

Bottom Section: Actual Transformer Model Implementation (Purple Boxes)

  1. Tokenization
    • String splitting, subword units, ID conversion, vocabulary mapping
  2. Embedding Feature
    • High-dimensional vector conversion, embedding matrix, semantic distance, placement in vector space
  3. Positional Encoding + Context Building
    • Positional information encoding, sine/cosine functions, context matrix, preserving sequence order
  4. Attention Mechanism
    • Query-Key-Value, attention scores, softmax weights, selective information extraction
  5. Feed Forward + Output
    • Non-linear transformation, 2-layer neural network, softmax probability distribution, next token prediction

Key Concept

This diagram maps traditional information retrieval concepts to modern transformer architecture implementations. It visualizes how abstract concepts in the top row are realized through concrete technical implementations in the bottom row, providing an educational resource for understanding how models like GPT and BERT work internally at each stage.

Summary

This diagram explains the end-to-end pipeline of transformer models by mapping traditional information retrieval concepts (distinction, classification, indexing, retrieval, processing) to technical implementations (tokenization, embedding, positional encoding, attention mechanism, feed-forward output). The top row shows abstract conceptual stages while the bottom row reveals the actual neural network components used in models like GPT and BERT. It serves as an educational bridge between high-level understanding and low-level technical architecture.

#NLP #TransformerModels #DeepLearning #Tokenization #AttentionMechanism #MachineLearning #AI #NeuralNetworks #GPT #BERT #PositionalEncoding #Embedding #InformationRetrieval #ArtificialIntelligence #DataScience

With Claude

Who is the first wall?

Posted on by lechuck park

AI Scaling: The 6 Major Bottlenecks (2025)

1. Data

  • High-quality text data expected to be depleted by 2026
  • Solutions: Synthetic data (fraud detection in finance, medical data), Few-shot learning

2. LLM S/W (Algorithms)

  • Ilya Sutskever: “The era of simple scaling is over. Now it’s about scaling the right things”
  • Innovation directions: Test-time compute scaling (OpenAI o1), Mixture-of-Experts architecture, Hybrid AI

3. Computing → Heat

  • GPT-3 training required 1,024 A100 GPUs for several months
  • By 2030, largest training runs projected at 2-45GW scale
  • GPU cluster heat generation makes cooling a critical challenge

4. Memory & Network ⚠️ Current Critical Bottleneck

Memory

  • LLMs grow 410x/2yr, computing power 750x/2yr vs DRAM bandwidth only 2x/2yr
  • HBM3E completely sold out for 2024-2025. AI memory market projected to grow at 27.5% CAGR

Network

  • Speed of light limitation causes tens to hundreds of ms latency over distance. Critical for real-time applications (autonomous vehicles, AR)
  • Large-scale GPU clusters require 800Gbps+, microsecond-level ultra-low latency

5. Power 💡 Long-term Core Constraint

  • Sam Altman: “The cost of AI will converge to the cost of energy. The abundance of AI will be limited by the abundance of energy”
  • Power infrastructure (transmission lines, transformers) takes years to build
  • Data centers projected to consume 7.5% of US electricity by 2030

6. Cooling

  • Advanced technologies like liquid cooling required. Infrastructure upgrades take 1+ year

“Who is the first wall?”

Critical Bottlenecks by Timeline:

  1. Current (2025): Memory bandwidth + Data quality
  2. Short-to-Mid term: Power infrastructure (5-10 years to build)
  3. Long-term: Physical limit of the speed of light

Summary

The “first wall” in AI scaling is not a single barrier but a multi-layered constraint system that emerges sequentially over time. Today’s immediate challenges are memory bandwidth and data quality, followed by power infrastructure limitations in the mid-term, and ultimately the fundamental physical constraint of the speed of light. As Sam Altman emphasized, AI’s future abundance will be fundamentally limited by energy abundance, with all bottlenecks interconnected through the computing→heat→cooling→power chain.

#AIScaling #AIBottleneck #MemoryBandwidth #HBM #DataCenterPower #AIInfrastructure #SpeedOfLight #SyntheticData #EnergyConstraint #AIFuture #ComputingLimits #GPUCluster #TestTimeCompute #MixtureOfExperts #SamAltman #AIResearch #MachineLearning #DeepLearning #AIHardware #TechInfrastructure

With Claude