feats(transformers):add seed_oss model

[iugoood]

Add

1 add seed_oss model
2 add UT

Due to excessive weight causing an out-of-memory (OOM) error on the NPU, only the first 12 layers are being tested.

Usage

from transformers import AutoTokenizer
from mindone.transformers import SeedOssForCausalLM
import mindspore as ms
model = SeedOssForCausalLM.from_pretrained("/mnt/disk3/ymc/Seed-OSS-36B-Instruct/")
tokenizer = AutoTokenizer.from_pretrained("/mnt/disk3/ymc/Seed-OSS-36B-Instruct/")

prompt = "Hey, are you conscious? Can you talk to me?"
inputs = tokenizer(prompt, return_tensors="np")

# Generate
generate_ids = model.generate(ms.tensor(inputs.input_ids), max_length=30)
tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]

Performance

Experiments are tested on Ascend Atlas 800T A2 machines with mindspore 2.6.0.

model	mode	speed
seed_oss	pynative	2.68 token/s

[gemini-code-assist]

Summary of Changes

Hello @iugoood, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the mindone/transformers library by integrating the new SeedOss model. It includes the full implementation of the SeedOss architecture, its associated task-specific model heads, and updates to the library's auto-loading mechanisms to support this new model. Additionally, a dedicated test suite has been added to validate the SeedOss model's functionality and ensure its seamless operation within the MindSpore framework.

Highlights

• New Model Integration: The SeedOss model, along with its various task-specific heads (CausalLM, QuestionAnswering, SequenceClassification, TokenClassification), has been added to the mindone/transformers library.
• Auto-Configuration Updates: The SeedOss model and its configuration (SeedOssConfig) are now registered within the auto-configuration and auto-modeling modules, enabling dynamic loading.
• Core Model Implementation: Detailed implementation of the SeedOss model architecture, including components like SeedOssRMSNorm, SeedOssMLP, SeedOssAttention, SeedOssDecoderLayer, and SeedOssRotaryEmbedding, is provided.
• Comprehensive Testing: A new test suite specifically for the SeedOss model has been introduced to ensure its correctness and compatibility.
• Test Infrastructure Refinement: The CausalLMModelTester has been updated, removing a parent argument from its constructor, likely to streamline testing for new models.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces the SeedOss model and its variants. The implementation appears solid and aligns with the library's existing patterns. However, I've identified a critical issue in a refactored test helper that could break other tests in the repository. Additionally, I've suggested an improvement to the test coverage for the new model and a minor style fix for better code maintainability.

[gemini-code-assist]

The parent argument has been removed from the __init__ method, but the create_and_check_model method in this class still references self.parent. This will cause an AttributeError when create_and_check_model is called.

Please refactor create_and_check_model to not depend on self.parent, or remove the method if it's no longer used by any tests. For instance, the method could return the values to be asserted, and the calling test could perform the assertion.

[gemini-code-assist]

For better code style and maintainability, the __all__ list should be sorted alphabetically.

Suggested change

	__all__ = [
	"SeedOssForCausalLM",
	"SeedOssForQuestionAnswering",
	"SeedOssPreTrainedModel",
	"SeedOssModel",
	"SeedOssForSequenceClassification",
	"SeedOssForTokenClassification",
	]
	__all__ = [
	"SeedOssForCausalLM",
	"SeedOssForQuestionAnswering",
	"SeedOssForSequenceClassification",
	"SeedOssForTokenClassification",
	"SeedOssModel",
	"SeedOssPreTrainedModel",
	]

[gemini-code-assist]

The test suite for the new SeedOss model currently only includes a test case for the base SeedOssModel. To ensure comprehensive testing, it's important to also verify the correctness of the model with task-specific heads, especially SeedOssForCausalLM.

Please consider adding another test case to SEEDOSS_CASES for SeedOssForCausalLM to validate its output against the PyTorch equivalent. This will improve test coverage and help catch potential issues with the LM head implementation.