New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add transformer engine fp8 attention #803

Draft

cli99 wants to merge 1 commit into mosaicml:main from cli99:te-attn-support

Contributor

cli99 commented Dec 13, 2023

This PR adds the TransformerEngine fp8 attention implemention. https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/attention.py.

To enable it, set the below fields in the yaml config.

precision: amp_fp8
model:
  attn_config:
    attn_type: te_multihead_attention
    kv_n_heads: 8
  fc_type: te


          add te fp8 attention

05b4ddd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet