Discussions

• 2 You must be logged in to vote

  🙏

  About the issue of interleaved pipeline scheduling

  lxgsbqylbk asked Apr 29, 2025 in Q&A · Closed · Unanswered

  

  

  0
• 2 You must be logged in to vote

  🙏

  Why are the moment values in the optimizer different after training with virtual_pipeline_size=1 or 2?

  lxgsbqylbk asked Apr 29, 2025 in Q&A · Closed · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  [QUESTION] When will model have _extra_state?

  1049451037 asked Jul 3, 2024 in Q&A · Unanswered

  

  

  

  

  2
• 1 You must be logged in to vote

  🙏

  How to convert Llama-2 huggingface checkpoint to the megatron format

  stale No activity in 60 days on issue or PR

  ken-arf asked Jan 10, 2024 in Q&A · Unanswered

  

  

  

  

  7
• 1 You must be logged in to vote

  🙏

  Propagating a tensor "v0" from First Layer to Subsequent Layers in Megatron with PP and TP

  shiroko98 asked Apr 9, 2025 in Q&A · Unanswered

  

  

  0
• 4 You must be logged in to vote

  🙏

  [QUESTION]　To convert a Llama 3.1 70B checkpoint in torch dcp format to the HuggingFace format,

  kaiyama12345679 asked Oct 19, 2024 in Q&A · Unanswered

  

  

  

  

  2
• 4 You must be logged in to vote

  💡

  [ENHANCEMENT] Training with variable sequence length

  YuvalAtir started Aug 28, 2024 in Ideas

  

  

  1
• 1 You must be logged in to vote

  🙏

  [BUG] selective activation recompute only decrease little of GPU memory usage during training

  bugm asked Oct 18, 2024 in Q&A · Answered

  

  

  

  

  17
• 2 You must be logged in to vote

  🙏

  [QUESTION] Error in pipeline parallelism training for multimodal model

  CFC87 asked Oct 15, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  GPU Memory changes by TP/PP and recompute-activations when the GPU-NUM is stable

  Listen-WLS asked Mar 17, 2025 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Results using pipeline parallel are inconsistent with model parallel

  JavaZeroo asked Mar 8, 2025 in Q&A · Closed · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Using Full recompute does not decrease vram usage

  JavaZeroo asked Mar 8, 2025 in Q&A · Unanswered

  

  

  0
• 5 You must be logged in to vote

  🙏

  [QUESTION] LLaVA model_type, pipeline parallel training

  KookHoiKim asked Aug 28, 2024 in Q&A · Unanswered

  

  

  1
• 2 You must be logged in to vote

  🙏

  [QUESTION] How to Obtain Computation Model Graphs in Megatron-LM?

  stale No activity in 60 days on issue or PR

  fwyc0573 asked May 19, 2024 in Q&A · Unanswered

  

  

  

  

  4
• 1 You must be logged in to vote

  🙏

  [QUESTION] Setting num-attention-heads=0 for Mamba

  zixianwang2022 asked Sep 8, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  Support for Heterogeneous Parallelism in Multimodal Training

  swiftomkar asked Feb 4, 2025 in Q&A · Unanswered

  

  

  0
• 3 You must be logged in to vote

  💡

  [ENHANCEMENT] Integrating torch.compile with Megatron/TransformerEngine

  yanboliang started Aug 20, 2024 in Ideas

  

  

  

  

  4
• 1 You must be logged in to vote

  🙏

  Flash attention support for megatron.core.transformer.attention.SelfAttention

  sinamoeini-amz asked Jan 21, 2025 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  [QUESTION]Where does the attention_mask come from when the gpt_model is not the first or last pipeline stage?

  janelu9 asked Jun 8, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  💬

  Selecting Which layers to recompute activations in?

  eliird started Dec 22, 2024 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  [QUESTION]I cannot figure out TE(transformer_engine)

  Guodanding asked Aug 29, 2024 in Q&A · Unanswered

  

  

  

  

  15
• 1 You must be logged in to vote

  🙏

  [QUESTION] Training Llama3 70B on 16 x A100 only achieves low throughput of 20 TFLOPS

  ZeroAGI asked Aug 13, 2024 in Q&A · Unanswered

  

  

  

  

  4
• 1 You must be logged in to vote

  🙏

  [QUESTION]Using FP8 OOM, otherwise --bf16 works well

  ghost asked Oct 23, 2024 in Q&A · Unanswered

  

  

  1
• 2 You must be logged in to vote

  🙏

  Does Megatron support tracing computation graphs with torch.fx?

  fy-j asked Dec 15, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  What git branch corresponds to the latest version and what Docker version to use?

  karim-manaouil asked Dec 4, 2024 in Q&A · Unanswered

  

  

  0