Tag: AIArchitecture

Prefill & Decode (2)

Posted on by lechuck park

1. Prefill Phase (Input Analysis & Parallel Processing)

  • Processing Method: It processes the user’s lengthy prompts or documents all at once using Parallel Computing.
  • Bottleneck (Compute-bound): Since it needs to process a massive amount of data simultaneously, computational power is the most critical factor. This phase generates the KV (Key-Value) cache which is used in the next step.
  • Requirements: Because it requires large-scale parallel computation, massive throughput and large-capacity memory are essential.
  • Hardware Characteristics: The image describes this as a “One-Hit & Big Size” style, explaining that a GPU-based HBM (High Bandwidth Memory) architecture is highly suitable for handling such large datasets.

2. Decode Phase (Sequential Token Generation)

  • Processing Method: Using the KV cache generated during the Prefill phase, this is a Sequential Computing process that generates the response tokens one by one.
  • Bottleneck (Memory-bound): The computation itself is light, but the system must constantly access the memory (KV cache) to fetch and generate the next word. Therefore, memory access speed (bandwidth) becomes the limiting factor.
  • Requirements: Because it needs to provide small and immediate responses to the user, ultra-low latency and deterministic execution speed are crucial.
  • Hardware Characteristics: Described as a “Small-Hit & Fast Size” style, an LPU (Language Processing Unit)-based SRAM architecture is highly advantageous to minimize latency.

💡Summary

  1. Prefill is a compute-bound phase that processes user input in parallel all at once to create a KV cache, making GPU and HBM architectures highly suitable.
  2. Decode is a memory-bound phase that sequentially generates words one by one by referencing the KV cache, where LPU and SRAM architectures are advantageous for achieving ultra-low latency.
  3. Ultimately, an LLM operates by grasping the context through large-scale computation (Prefill) and then generating responses in real-time through fast memory access (Decode).

#LLM #Prefill #Decode #GPU_HBM #LPU_SRAM #AIArchitecture #ParallelComputing #UltraLowLatencyAI

With Gemini

vLLM Features

Posted on by lechuck park

vLLM Features & Architecture Breakdown

This chart outlines the key components of vLLM (Virtual Large Language Model), a library designed to optimize the inference speed and memory efficiency of Large Language Models (LLMs).

1. Core Algorithm

  • PagedAttention
    • Concept: Applies the operating system’s (OS) virtual memory paging mechanism to the attention mechanism.
    • Benefit: It resolves memory fragmentation and enables the storage of the KV (Key-Value) cache in non-contiguous memory spaces, significantly reducing memory waste.

2. Data Unit

  • Block (Page)
    • Concept: The minimum KV cache unit with a fixed token size (e.g., 16 tokens).
    • Benefit: Increases management efficiency via fixed-size allocation and minimizes wasted space (internal fragmentation) within slots.
  • Block Table
    • Concept: A mapping table that connects Logical Blocks to Physical Blocks.
    • Benefit: Allows non-contiguous physical memory to be processed as if it were a continuous context.

3. Operation

  • Pre-allocation (Profiling)
    • Concept: Reserves the maximum required VRAM at startup by running a dummy simulation.
    • Benefit: Eliminates the overhead of runtime memory allocation/deallocation and prevents Out Of Memory (OOM) errors at the source.

4. Memory Handling

  • Swapping
    • Concept: Offloads data to CPU RAM when GPU memory becomes full.
    • Benefit: Handles traffic bursts without server downtime and preserves the context of suspended (waiting) requests.
  • Recomputation
    • Concept: Recalculates data instead of swapping it when recalculation is more cost-effective.
    • Benefit: Optimizes performance for short prompts or in environments with slow interconnects (e.g., PCIe limits).

5. Scheduling

  • Continuous Batching
    • Concept: Iteration-level scheduling that fills idle slots immediately without waiting for other requests to finish.
    • Benefit: Eliminates GPU idle time and maximizes overall throughput.

Summary

  1. vLLM adapts OS memory management techniques (like Paging and Swapping) to optimize LLM serving, solving critical memory fragmentation issues.
  2. Key technologies like PagedAttention and Continuous Batching minimize memory waste and eliminate GPU idle time to maximize throughput.
  3. This architecture ensures high performance and stability by preventing memory crashes (OOM) and efficiently handling traffic bursts.

#vLLM #LLMInference #PagedAttention #AIArchitecture #GPUOptimization #MachineLearning #SystemDesign #AIInfrastructure

With Gemini

Basic LLM Workflow

Posted on by lechuck park

Basic LLM Workflow Interpretation

This diagram illustrates how data flows through various hardware components during the inference process of a Large Language Model (LLM).

Step-by-Step Breakdown

① Initialization Phase (Warm weights)

  • Model weights are loaded from SSD → DRAM → HBM (High Bandwidth Memory)
  • Weights are distributed and shared across multiple GPUs

② Input Processing (CPU tokenizes/batches)

  • CPU tokenizes input text and processes batches
  • Data is transferred through DRAM buffer to GPU

③ GPU Inference Execution

  • GPU performs Attention and FFN (Feed-Forward Network) computations from HBM
  • KV cache (Key-Value cache) is stored in HBM
  • If HBM is tight, KV cache can be offloaded to DRAM or SSD

④ Distributed Communication (NvLink/Infiniband)

  • Intra-node: High-speed communication between GPUs via NvLink (with NVSwitch if available)
  • Inter-node: Parallel communication through InfiniBand or NCCL

⑤ Post-processing (CPU decoding/post)

  • CPU decodes generated tokens and performs post-processing
  • Logs and caches are saved to SSD

Key Characteristics

This architecture leverages a memory hierarchy to efficiently execute large-scale models:

  • SSD: Long-term storage (slowest, largest capacity)
  • DRAM: Intermediate buffer
  • HBM: GPU-dedicated high-speed memory (fastest, limited capacity)

When model size exceeds GPU memory, strategies include distributing across multiple GPUs or offloading data to higher-level memory tiers.

Summary

This diagram shows how LLMs process data through a memory hierarchy (SSD→DRAM→HBM) across CPU and GPU components. The workflow involves loading model weights, tokenizing inputs on CPU, running inference on GPU with HBM, and using distributed communication (NvLink/InfiniBand) for multi-GPU setups. Memory management strategies like KV cache offloading enable efficient execution of large models that exceed single GPU capacity.

#LLM #DeepLearning #GPUComputing #MachineLearning #AIInfrastructure #NeuralNetworks #DistributedComputing #HPC #ModelOptimization #AIArchitecture #NvLink #Transformer #MLOps #AIEngineering #ComputerArchitecture

With Claude