DeepSeekV3技术报告

DeepSeek-V3TechnicalReportDeepSeek-AIresearch@deepseek.comAbstractWepresentDeepSeek-V3,astrongMixture-of-Experts(MoE)languagemodelwith671Btotalparameterswith37Bactivatedforeachtoken.Toachieveefficientinferenceandcost-effectivetraining,DeepSeek-V3adoptsMulti-headLatentAttention(MLA)andDeepSeekMoEarchitec-tures,whichwerethoroughlyvalidatedinDeepSeek-V2.Furthermore,DeepSeek-V3pioneersanauxiliary-loss-freestrategyforloadbalancingandsetsamulti-tokenpredictiontrainingobjectiveforstrongerperformance.Wepre-trainDeepSeek-V3on14.8trilliondiverseandhigh-qualitytokens,followedbySupervisedFine-TuningandReinforcementLearningstagestofullyharnessitscapabilities.ComprehensiveevaluationsrevealthatDeepSeek-V3outperformsotheropen-sourcemodelsandachievesperformancecomparabletoleadingclosed-sourcemodels.Despiteitsexcellentperformance,DeepSeek-V3requiresonly2.788MH800GPUhoursforitsfulltraining.Inaddition,itstrainingprocessisremarkablystable.Throughouttheentiretrainingprocess,wedidnotexperienceanyirrecoverabl