SambaNovaSN40LRDU突破万亿参数规模GenAI计算的壁垒

SambaNova SN40L RDU: Breaking the Barrier of Trillion+ Parameter Scale Gen AI ComputingRaghu PrabhakarArchitect, SambaNovaSystemsHotChips2024Copyright © 2024 SambaNova Systems Inc.SN40L:SambaNova’sNew Language-Optimized RDU2“Cerulean” Architecture-based Reconfigurable Dataflow Unit1.5 TB High Capacity Memory5nm TSMC3-tier Dataflow Memory1,040 RDU Cores102B Transistors64 GB High Bandwidth Memory520 MB On-Chip Memory638 TFLOPS (bf16)Cerulean SN40L RDUGenerative AI Training and InferenceCopyright © 2024 SambaNova Systems Inc.On-Chip SRAM[8 GB, PBs per sec]RDU High Bandwidth Memory [1 TB] RDU High Capacity DDR Memory [24 TB]1600 GB/s25.6 TB/sHigh throughput inference with caching Low Latency Model Switching(E.g., < 0.01s for llama3.1 8B)Dataflow enabled by large On-Chip Memory3SN40L:SambaNova’sNew Language-Optimized RDU3-tier Memory System with SRAM, HBM, and DDRCopyright © 2024 SambaNova Systems Inc.SN40L Chip: Tile Architecture 1040PCUs and PMUsPCU: Compute unitPMU: Memory unitS: Mesh sw