2604.12374v1
Apr 14, 2026
cs.LG
Nemotron 3 Super: 에이전트 기반 추론을 위한 개방형, 효율적인 Mamba-Transformer 혼합 전문가 모델
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
M. Patwary
M. Patwary
Citations:
8,251
h-index:
28
M. Shoeybi
M. Shoeybi
Citations:
15,265
h-index:
45
Itay Levy
Itay Levy
NVIDIA
Citations:
325
h-index:
5
Omri Puny
Omri Puny
Citations:
570
h-index:
9
Jiaqi Zeng
Jiaqi Zeng
Citations:
1,033
h-index:
12
Fei Jia
Fei Jia
Citations:
406
h-index:
6
V. Noroozi
V. Noroozi
Citations:
2,585
h-index:
19
Dan Su
Dan Su
Citations:
589
h-index:
10
S. Velury
S. Velury
Citations:
103
h-index:
3
K. Luna
K. Luna
Citations:
311
h-index:
7
E. Long
E. Long
Citations:
406
h-index:
9
Ann Guan
Ann Guan
Citations:
233
h-index:
6
Grace Lam
Grace Lam
Citations:
3,318
h-index:
8
Jan Kautz
Jan Kautz
Citations:
907
h-index:
5
Sadegh Mahdavi
Sadegh Mahdavi
University of British Columbia
Citations:
233
h-index:
8
H. Kim
H. Kim
Citations:
44
h-index:
4
V. Elango
V. Elango
Citations:
612
h-index:
12
J. Guman
J. Guman
Citations:
84
h-index:
4
B. Rouhani
B. Rouhani
Citations:
2,705
h-index:
22
Dongfu Jiang
Dongfu Jiang
University of Waterloo
Citations:
4,100
h-index:
14
N. Qin
N. Qin
Citations:
12
h-index:
2
Ouye Xie
Ouye Xie
Citations:
79
h-index:
3
Ritu Gala
Ritu Gala
Citations:
146
h-index:
4
Liang Ding
Liang Ding
University of Sydney / JD Explore Academy
Citations:
5,343
h-index:
40
A. Ahamed
A. Ahamed
Citations:
101
h-index:
4
B. Simkin
B. Simkin
Citations:
156
h-index:
5
Ben Lanir
Ben Lanir
Citations:
136
h-index:
5
Brian Yu
Brian Yu
Citations:
10
h-index:
1
Chao Ni
Chao Ni
Citations:
38
h-index:
3
C. Wing
C. Wing
Citations:
15
h-index:
2
C. Desai
C. Desai
Citations:
11
h-index:
2
D. Egert
D. Egert
Citations:
20
h-index:
2
D. Lo
D. Lo
Citations:
89
h-index:
4
Dong Ahn
Dong Ahn
Citations:
268
h-index:
6
E. Lewis
E. Lewis
Citations:
9
h-index:
1
Evan Wu
Evan Wu
Citations:
77
h-index:
4
Fay Wang
Fay Wang
Citations:
78
h-index:
3
G. Prasad
G. Prasad
Citations:
262
h-index:
6
Grace Wu
Grace Wu
Citations:
292
h-index:
4
G. Davis
G. Davis
Citations:
9
h-index:
1
H. Kumar
H. Kumar
Citations:
10
h-index:
1
Hayate Iso
Hayate Iso
NVIDIA
Citations:
423
h-index:
13
I. Schen
I. Schen
Citations:
57
h-index:
4
J. Tu
J. Tu
Citations:
9
h-index:
1
J. Casper
J. Casper
Citations:
716
h-index:
9
J. Sewall
J. Sewall
Citations:
1,941
h-index:
21
Jie Lou
Jie Lou
Citations:
132
h-index:
6
Jim Chow
Jim Chow
Citations:
2,764
h-index:
9
J. Cohen
J. Cohen
Citations:
595
h-index:
6
Juan Yu
Juan Yu
Citations:
61
h-index:
3
J. Tapia
J. Tapia
Citations:
14
h-index:
2
Luis Vega
Luis Vega
Citations:
217
h-index:
5
Mark Cai
Mark Cai
Citations:
143
h-index:
4
M. Gabel
M. Gabel
Citations:
1,410
h-index:
9
Meng Xin
Meng Xin
Citations:
24
h-index:
3
M. Wang
M. Wang
Citations:
63
h-index:
3
N. Habibi
N. Habibi
Citations:
308
h-index:
7
Nir Ailon
Nir Ailon
Citations:
5,906
h-index:
29
N. Pitt
N. Pitt
Citations:
9
h-index:
1
Peter Jin
Peter Jin
Citations:
151
h-index:
5
Qiyu Wan
Qiyu Wan
Citations:
76
h-index:
3
Q. Minh
Q. Minh
Citations:
35
h-index:
3
R. Zhong
R. Zhong
Citations:
9
h-index:
1
R. Yao
R. Yao
Citations:
24
h-index:
2
R. Wang
R. Wang
Citations:
36
h-index:
2
R. Clark
R. Clark
Citations:
14
h-index:
2
Ruisi Cai
Ruisi Cai
Citations:
1,035
h-index:
11
R. Prenger
R. Prenger
Citations:
10,360
h-index:
19
S. Pombra
S. Pombra
Citations:
276
h-index:
4
S. Pombo
S. Pombo
Citations:
40
h-index:
3
S. Hara
S. Hara
Citations:
29
h-index:
3
Sepehr Sameni
Sepehr Sameni
University of Bern
Citations:
179
h-index:
6
Shantanu Acharya
Shantanu Acharya
New York University
Citations:
1,284
h-index:
9
S. Lord
S. Lord
Citations:
9
h-index:
1
T. Yuan
T. Yuan
Citations:
10
h-index:
1
T. Bui
T. Bui
Citations:
14
h-index:
2
T. Do
T. Do
Citations:
9
h-index:
1
Tim Moon
Tim Moon
Citations:
143
h-index:
4
V. Shah
V. Shah
Citations:
90
h-index:
4
Feng Lin
Feng Lin
Citations:
154
h-index:
3
Di Wu
Di Wu
Citations:
115
h-index:
3
A. Liu
A. Liu
Citations:
9
h-index:
1
A. Jain
A. Jain
Citations:
21
h-index:
2
Chen Luo
Chen Luo
Citations:
18
h-index:
2
Jun Yi
Jun Yi
Citations:
33
h-index:
4
Leo Du
Leo Du
Citations:
9
h-index:
1
Pinky Xu
Pinky Xu
Citations:
78
h-index:
3
S. Thomas
S. Thomas
Citations:
262
h-index:
6
T. Ma
T. Ma
Citations:
22
h-index:
2
T. Konuk
T. Konuk
Citations:
466
h-index:
9
B. Mani
B. Mani
Citations:
71
h-index:
5
Andrew Tao
Andrew Tao
Citations:
12,624
h-index:
31
본 논문에서는 1200억 개의 파라미터(활성 파라미터 120억 개)를 가지는 Mamba-Attention 혼합 전문가 모델인 Nemotron 3 Super의 사전 훈련, 추가 훈련 및 양자화 과정을 설명합니다. Nemotron 3 Super는 Nemotron 3 제품군에서 처음으로 1) NVFP4에서 사전 훈련되었으며, 2) FLOP당 정확도와 파라미터당 정확도를 모두 최적화하는 새로운 혼합 전문가 아키텍처인 LatentMoE를 활용하고, 3) 네이티브 추론 가속을 위한 MTP 레이어를 포함합니다. Nemotron 3 Super는 25조 개의 토큰으로 사전 훈련을 수행한 후, 지도 학습 미세 조정(SFT) 및 강화 학습(RL)을 통해 추가 훈련을 진행했습니다. 최종 모델은 최대 1M의 컨텍스트 길이를 지원하며, 일반적인 벤치마크에서 GPT-OSS-120B 및 Qwen3.5-122B와 비교하여 동등한 정확도를 달성하는 동시에, 추론 처리량을 각각 최대 2.2배 및 7.5배 향상시켰습니다. Nemotron 3 Super의 데이터셋과 함께, 기본 모델, 추가 훈련된 모델 및 양자화된 모델의 체크포인트는 HuggingFace에서 공개됩니다.
Original
Abstract
We describe the pre-training, post-training, and quantization of Nemotron 3 Super, a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. Nemotron 3 Super is the first model in the Nemotron 3 family to 1) be pre-trained in NVFP4, 2) leverage LatentMoE, a new Mixture-of-Experts architecture that optimizes for both accuracy per FLOP and accuracy per parameter, and 3) include MTP layers for inference acceleration through native speculative decoding. We pre-trained Nemotron 3 Super on 25 trillion tokens followed by post-training using supervised fine tuning (SFT) and reinforcement learning (RL). The final model supports up to 1M context length and achieves comparable accuracy on common benchmarks, while also achieving up to 2.2x and 7.5x higher inference throughput compared to GPT-OSS-120B and Qwen3.5-122B, respectively. Nemotron 3 Super datasets, along with the base, post-trained, and quantized checkpoints, are open-sourced on HuggingFace.
Original PDF
{ copied = true; setTimeout(() => copied = false, 2000); })"
class="btn btn-square btn-sm btn-ghost text-base-content/50 hover:text-primary hover:bg-base-200 transition-colors mb-1"
:class="{ 'tooltip tooltip-success': copied, 'tooltip': !copied }"
:data-tip="copied ? '복사 완료!' : 'PDF 링크 복사'">
Copy PDF Link
No Analysis Report Yet
This paper hasn't been analyzed by Gemini yet.
Log in to request an AI analysis.