Support llama3 eagle3 head with llama4 verifier

[rahul-tuli]

Summary

This PR enables Eagle3 speculative decoding with Llama3 drafter and Llama4 multimodal verifier support, with configurable auxiliary hidden state layers.

Key Features

• Compatibility: Llama3 Eagle3 drafter can now work with Llama4 verifier models
• Configurable auxiliary layers: Hidden state layer indices can be specified via eagle_aux_hidden_state_layer_ids in the speculator config, allowing non-default layer selection for optimal performance across different model architectures

Configuration

Auxiliary layer indices can be set in the Eagle3 draft model config:

{
  "eagle_aux_hidden_state_layer_ids": [1, 23, 44]
}

This enables using hidden states from non-default layers (e.g., layers 1, 23, 44 instead of default 2, 23, 44) for cross-architecture scenarios where different layer combinations may work better.

Testing

Command:

python examples/offline_inference/spec_decode.py \
  --method "eagle3" \
  --tp 8 \
  --print-output \
  --model-dir "RedHatAI/Llama-4-Maverick-17B-128E-Instruct-quantized.w4a16" \
  --eagle-dir "nm-testing/Llama4-Maverick-Eagle3-Speculators" \
  --dataset_name "hf" \
  --dataset_path "philschmid/mt-bench" \
  --num-spec-tokens 3

Results:

• Mean acceptance length: 2.53
• Per-position acceptance rates: 0.71, 0.48, 0.34
• Auxiliary layers used: [1, 23, 44] (configured via speculator config)

--------------------------------------------------
--------------------------------------------------
total_num_output_tokens: 227215
num_drafts: 90393
num_draft_tokens: 271179
num_accepted_tokens: 136677
mean acceptance length: 2.53
--------------------------------------------------
acceptance at token 0: 0.71
acceptance at token 1: 0.48
acceptance at token 2: 0.34