Skip to content

Comments

MOE implementation#5

Open
MihirsinhChauhan wants to merge 1 commit intoCohleM:masterfrom
MihirsinhChauhan:master
Open

MOE implementation#5
MihirsinhChauhan wants to merge 1 commit intoCohleM:masterfrom
MihirsinhChauhan:master

Conversation

@MihirsinhChauhan
Copy link
Contributor

Changes

  • Added new MoEFFN class for expert-based feed-forward networks
  • Modified TransformerBlock to use MoE instead of standard FFN
  • Added MoE configuration parameters to model config

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant