Skip to content
/ GsiT Public

Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphs

License

Notifications You must be signed in to change notification settings

drewjin/GsiT

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

18 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphs

Acknowledgement

The repository is based on MMSA.

We strongly recommend integrating the core code directly into the MMSA framework. This repository is structured entirely in line with the MMSA setup. However, minor issues may still arise, and we suggest directly incorporating the files from this repository into the MMSA framework for seamless execution.

Main Components

The main model is in GsiT/src/MMSA-GsiT/models/custom/GSIT/.

The main model trainer is in GsiT/src/MMSA-GsiT/trains/custom/GSIT.py.

The main configuration is in GsiT/src/MMSA-GsiT/config/config_regression.json.

The Triton kernel is in GsiT/src/MMSA-GsiT/models/custom/GSIT/modules/Kernel.

About

Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphs

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published