Skip to content

add_axk1 - #46867

Merged
vasqu merged 19 commits into
huggingface:mainfrom
kmswin1:feature/add_axk1
Jul 23, 2026
Merged

add_axk1#46867
vasqu merged 19 commits into
huggingface:mainfrom
kmswin1:feature/add_axk1

Conversation

@kmswin1

@kmswin1 kmswin1 commented Jun 24, 2026 •

Copy link
Copy Markdown
Contributor

CI

Add A.X K1 model architecture
What does this PR do?
This PR adds support for A.X K1, a large-scale Mixture-of-Experts (MoE) language model developed by SK Telecom. A.X K1 contains 519B total parameters with 33B active parameters per token, enabling strong reasoning performance with practical inference efficiency.

Paper: A.X K1 Technical Report (arXiv:2601.09200)
Model: skt/A.X-K1
License: Apache 2.0

Architecture overview
A.X K1 is a decoder-only Transformer with the following key design choices:

Sparse MoE: 61 layers (1 dense + 60 MoE), with 192 routed experts + 1 shared expert per MoE layer. 8 experts + 1 shared expert are activated per token.
Multi-Latent Attention (MLA): Compressed KV representations for efficient long-context inference.
Post-MLP RMSNorm: An additional RMSNorm applied after the MLP/MoE block in each layer, improving training stability in large-scale sparse settings.
Multi-Token Prediction (MTP): An auxiliary training objective that also enables speculative decoding at inference time.
Hybrid Reasoning (Think / Non-Think): Supports user-controllable reasoning depth within a single unified model.
Context length: 131,072 tokens
Vocab size: 163,840 (BBPE-based tokenizer optimized for EN, KO, ZH, JA, ES and code)

This PR includes

Model implementation (AXK1ForCausalLM, AXK1Model, AXK1Config)
Tokenizer integration (BBPE-based, 163,840 vocab)
Documentation and model card
Tests

Before submitting

I confirm that this is not a pure code agent PR.
This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline, Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?
@ArthurZucker @Cyrilvallez β€” text model architecture addition

@LysandreJik

Copy link
Copy Markdown
Member

Thanks @kmswin1, we're taking a look

@kmswin1
kmswin1 force-pushed the feature/add_axk1 branch from acb99b0 to 89848e1 Compare June 30, 2026 14:14

@Cyrilvallez Cyrilvallez left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey! Sorry for the delay! Very nice first implem in general!
We need to make sure that we follow naming conventions, i.e. snake_case for model_type, filenames etc, and CamelCase for classes though, no uppercases.
Missing tests as well!

Comment on lines +50 to +51
("autoformer", "AutoformerConfig"),
("AXK1", "AXK1Config"),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using snake_case and CamelCase conventions in Transformers!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your review!

Our model name is A.X-K Series (e.g. not A.x-k1), and our standard convention is to use uppercase. Could we maintain this naming rule?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If possible, we would prefer to keep the uppercase naming to stay consistent with the official A.X-K Series naming. That said, I understand the library convention and don’t want this to block the PR. If keeping the uppercase form is not acceptable, I’m happy to update the internal model_type, module names, and registry keys to lowercase/snake_case, e.g. axk1, while keeping the public class names as AXK1Config / AXK1Model.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, let's do that then!

Comment on lines +425 to +427
("sam3_lite_text", "Sam3LiteTextConfig"),
("sam3_lite_text_text_model", "Sam3LiteTextTextConfig"),
("sam3_tracker", "Sam3TrackerConfig"),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this??

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover from an unrelated local branch removed. The auto mappings now only add the AXK1 entries.

Comment on lines +961 to +962
("sam3_lite_text", "SAM3-LiteText"),
("sam3_lite_text_text_model", "SAM3-LiteText Text Model"),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

??

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover from an unrelated local branch removed. The auto mappings now only add the AXK1 entries.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover from an unrelated local branch removed. The auto mappings now only add the AXK1 entries.

)

# This is tied to the processing `-` -> `_` in `model_type_to_module_name`. For example, instead of putting
# `transfo-xl` (as in `CONFIG_MAPPING_NAMES`), we should use `transfo_xl`.
DEPRECATED_MODELS = []

SPECIAL_MODEL_TYPE_TO_MODULE_NAME = OrderedDict[str, str](
[
("AXK1", "AXK1"),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said, we really want to fllow snake_case and CamelCase conventions

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged. we'd like to keep the AXK1 casing for now since the released checkpoints on the Hub use model_type: "AXK1" and renaming would break AutoConfig for existing configs. Happy to migrate to axk1/Axk1* in a follow-up if you feel strongly; let us know your preference for handling the existing checkpoints.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the model_type itself should really be in snake_case - it's very easy to change the config directly on the hub, just a one-line change!

Comment on lines +20 to +21

AXK1_PRETRAINED_CONFIG_ARCHIVE_MAP = {}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To remove

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment on lines +119 to +120

class AXK1NaiveMoe(MixtralExperts):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
class AXK1NaiveMoe(MixtralExperts):
class Axk1Experts(MixtralExperts):

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is tied to the casing discussion above.

Comment on lines +147 to +173
def route_tokens_to_experts(self, router_logits):
router_logits = router_logits.sigmoid()
if self.gate.e_score_correction_bias is not None:
router_logits_for_choice = router_logits + self.gate.e_score_correction_bias
else:
router_logits_for_choice = router_logits
group_scores = (
router_logits_for_choice.view(-1, self.n_group, self.n_routed_experts // self.n_group)
.topk(2, dim=-1)[0]
.sum(dim=-1)
)
group_idx = torch.topk(group_scores, k=self.topk_group, dim=-1, sorted=False)[1]
group_mask = torch.zeros_like(group_scores)
group_mask.scatter_(1, group_idx, 1)
score_mask = (
group_mask.unsqueeze(-1)
.expand(-1, self.n_group, self.n_routed_experts // self.n_group)
.reshape(-1, self.n_routed_experts)
)
scores_for_choice = router_logits_for_choice.masked_fill(~score_mask.bool(), 0.0)
topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
topk_weights = router_logits.gather(1, topk_indices)
if self.norm_topk_prob:
denominator = topk_weights.sum(dim=-1, keepdim=True) + 1e-20
topk_weights /= denominator
topk_weights = topk_weights * self.routed_scaling_factor
return topk_indices, topk_weights

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be part of router's forward!

@kmswin1 kmswin1 Jul 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Routing (sigmoid, group top-k, normalization, scaling) now lives in AXK1TopkRouter.forward, which returns (topk_indices, topk_weights); AXK1MoE is a thin wrapper.

Comment on lines +266 to +269
if self.config.rope_interleave:
q_rot, k_rot = apply_rotary_pos_emb_interleave(q_rot, k_rot, cos, sin)
else:
q_rot, k_rot = apply_rotary_pos_emb(q_rot, k_rot, cos, sin)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we ever have both paths in released checkpoints?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. all released checkpoints use the interleaved layout, so the flag and the non-interleaved branch are removed; the model always applies interleaved RoPE.

Comment on lines +323 to +324
self.input_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
self.post_attention_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
self.input_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
self.post_attention_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
self.input_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
self.post_attention_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps) is not self.is_moe_layer else nn.Identity()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: dense layers get nn.Identity() and the branch in the decoder forward is gone.

Comment on lines +327 to +357
def forward(
self,
hidden_states: torch.Tensor,
attention_mask: torch.Tensor | None = None,
position_ids: torch.LongTensor | None = None,
past_key_values: Cache | None = None,
use_cache: bool | None = False,
position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
**kwargs: Unpack[TransformersKwargs],
) -> torch.Tensor:
residual = hidden_states
hidden_states = self.input_layernorm(hidden_states)

hidden_states, _ = self.self_attn(
hidden_states=hidden_states,
attention_mask=attention_mask,
position_ids=position_ids,
past_key_values=past_key_values,
use_cache=use_cache,
position_embeddings=position_embeddings,
**kwargs,
)
hidden_states = residual + hidden_states

residual = hidden_states
hidden_states = self.post_attention_layernorm(hidden_states)
hidden_states = self.mlp(hidden_states)
if self.is_moe_layer:
hidden_states = self.post_mlp_layernorm(hidden_states)
hidden_states = residual + hidden_states
return hidden_states

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To remove, with the previous comment about Identity

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: the branch is removed; post_mlp_layernorm is now applied unconditionally (nn.Identity on dense layers).

@kmswin1

kmswin1 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Hey! Sorry for the delay! Very nice first implem in general! We need to make sure that we follow naming conventions, i.e. snake_case for model_type, filenames etc, and CamelCase for classes though, no uppercases. Missing tests as well!

Thank you for the review! I’ll go through the comments one by one and update the PR accordingly.

@kmswin1

kmswin1 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Thanks again for the thorough review! I addressed the comments in the latest commit:

  • Removed the unrelated sam3_lite_text leftovers and the empty ARCHIVE_MAP.
  • Moved the routing logic into AXK1TopkRouter.forward, which now returns the selected indices and weights. The e_score_correction_bias logic is also handled there, and AXK1MoE is now a thin wrapper around the router and experts.
  • Inlined yarn_get_mscale.
  • Imported apply_rotary_pos_emb_interleave from deepseek_v3. I kept the Cohere helper out because its interleaved RoPE convention is not numerically equivalent in our setting.
  • Removed the unused non-interleaved RoPE path and the rope_interleave flag, since all released checkpoints use the interleaved path.
  • Changed post_mlp_layernorm to nn.Identity for dense layers and removed the corresponding decoder forward branch.

I also verified that this refactor preserves behavior: generation with the released checkpoint is unchanged after the update.

The only remaining open point is the AXK1 casing. I kept it for now because the released Hub checkpoints currently ship with model_type: "AXK1", but I’m happy to migrate this to lowercase/snake_case in a follow-up if that is preferred.

@Cyrilvallez

Copy link
Copy Markdown
Member

Hey @kmswin1! Alright, let's go with the tradeoff of model_type in snake_case, while keeping ModuleClasses as AXK1Module then!
Also, please rebase/merge the main branch to fix conflicts
Finally, don't forget to add a test folder for the model, as well as a few integrations tests on real checkpoints! An example of this can be seen in test_modeling_llama.py (where the fast tests class LlamaModelTest(CausalLMModelTest, unittest.TestCase) inherits from CausalLMModelTest, and the IntegrationTests, or slow tests, are tests on real checkpoints)

@Cyrilvallez

Copy link
Copy Markdown
Member

Ping me once all this is done for a new review!

kmswin1 added 2 commits July 9, 2026 16:30
# Conflicts:
#	src/transformers/models/auto/configuration_auto.py
#	src/transformers/utils/auto_docstring.py
@kmswin1

kmswin1 commented Jul 9, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks again! @Cyrilvallez

I’ve updated the PR accordingly:

  • Changed the internal model_type / registry names to snake_case, e.g. axk1, while keeping the public class names as AXK1Config, AXK1Model, and AXK1ForCausalLM.
  • Rebased on the latest main and resolved the conflicts.
  • Added the model test folder with fast modeling tests.
  • Added slow integration tests on the released checkpoint, following the structure used in test_modeling_llama.py.

Please let me know if there is anything else you’d like me to adjust.

@vasqu vasqu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments from my side πŸ€—

Comment on lines +74 to +77
if getattr(config, "topk_method", "noaux_tc") == "noaux_tc":
self.register_buffer("e_score_correction_bias", torch.zeros(self.n_routed_experts))
else:
self.e_score_correction_bias = None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there really models that use this option? In any case we can still inherit from modular

  • The init can be used with super and only the e score needs to change
  • For the forward, we rewrite but add a small comment for the exact change

self.intermediate_dim = config.moe_intermediate_size


class AXK1MoE(nn.Module):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty sure that e.g. deepseek v2 has a similar strucute but you need to sync with main

return hidden_states


class AXK1Attention(nn.Module):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks very deepseek v3 ish? Let's use modular please

Comment on lines +258 to +281
nn.Module.__init__(self)
self.config = config
self.layer_idx = layer_idx
self.hidden_size = config.hidden_size

self.self_attn = AXK1Attention(config=config, layer_idx=layer_idx)

self.is_moe_layer = (
config.n_routed_experts is not None
and layer_idx >= config.first_k_dense_replace
and layer_idx % config.moe_layer_freq == 0
)
if self.is_moe_layer:
self.mlp = AXK1MoE(config)
else:
self.mlp = AXK1MLP(config)

self.input_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
self.post_attention_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
# `post_mlp_layernorm` only exists on MoE layers.
if self.is_moe_layer:
self.post_mlp_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
else:
self.post_mlp_layernorm = nn.Identity()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
nn.Module.__init__(self)
self.config = config
self.layer_idx = layer_idx
self.hidden_size = config.hidden_size
self.self_attn = AXK1Attention(config=config, layer_idx=layer_idx)
self.is_moe_layer = (
config.n_routed_experts is not None
and layer_idx >= config.first_k_dense_replace
and layer_idx % config.moe_layer_freq == 0
)
if self.is_moe_layer:
self.mlp = AXK1MoE(config)
else:
self.mlp = AXK1MLP(config)
self.input_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
self.post_attention_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
# `post_mlp_layernorm` only exists on MoE layers.
if self.is_moe_layer:
self.post_mlp_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
else:
self.post_mlp_layernorm = nn.Identity()
super().__init__(config, layer_idx)
self.mlp = AXK1MoE(config) if config.mlp_layer_types[layer_idx] == "sparse" else AXK1MLP(config)
self.post_attention_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps) if config.mlp_layer_types[layer_idx] == "sparse" else nn.Identity()

let's use mlp layer types instead please


class AXK1PreTrainedModel(LlamaPreTrainedModel):
_keep_in_fp32_modules_strict = ["e_score_correction_bias"]
_keys_to_ignore_on_load_unexpected = [r"model\.layers\.\d+\.self_attn\.rotary_emb\.inv_freq"]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
_keys_to_ignore_on_load_unexpected = [r"model\.layers\.\d+\.self_attn\.rotary_emb\.inv_freq"]
_keys_to_ignore_on_load_unexpected = [r"inv_freq"]

  - inherit from deepseek_v3 (AXK1Attention / AXK1TopkRouter / AXK1MoE / AXK1Experts /
    AXK1DecoderLayer / AXK1Model / AXK1ForCausalLM), keeping the A.X-K1-specific
    post_mlp_layernorm (applied to the MoE block output) as the only override
  - config: rewrite in @strict style inheriting DeepseekV3Config (class-level typed
    attributes, automatic docstring), moved into modular_axk1.py; drop pretraining_tp /
    moe_layer_freq / the rope-float override; inherit base_model_ep_plan
  - experts AXK1NaiveMoe(MixtralExperts) -> AXK1Experts(DeepseekV3Experts) (same stacked
    layout / keys); routing now lives in the router
  - _keys_to_ignore_on_load_unexpected = ["inv_freq"]
  - add docs (model_doc/axk1.md + toctree)
  - bump copyrights to 2026

  Verified on the light checkpoint: loads with the only newly-initialized key being the
  (zero) e_score_correction_bias the checkpoint omits, and greedy generation is coherent
  ("λŒ€ν•œλ―Όκ΅­μ˜ μˆ˜λ„λŠ”" -> " μ„œμšΈμ΄λ‹€.").
@kmswin1

kmswin1 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@vasqu

I have also applied the relevant changes discussed during the A.X-K2 review to A.X-K1, including the updated modular/config patterns, naming conventions, documentation, tests, and repository alignment where applicable.

@vasqu vasqu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small comments left. One important thing tho: We need tests, e.g. following deepseek v3. I think youtu is the best example because it uses our new structure for tests: https://github.com/huggingface/transformers/blob/main/tests/models/youtu/test_modeling_youtu.py

Comment thread docs/source/en/model_doc/axk1.md Outdated
Comment on lines +47 to +48
dtype=torch.bfloat16,
device_map="auto",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
dtype=torch.bfloat16,
device_map="auto",

Comment thread docs/source/en/model_doc/axk1.md Outdated
tokenizer = AutoTokenizer.from_pretrained("skt/A.X-K1")
model = AutoModelForCausalLM.from_pretrained(
"skt/A.X-K1",
dtype=torch.bfloat16,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
dtype=torch.bfloat16,

Comment on lines +59 to +60
first_k_dense_replace (`int`, *optional*, defaults to 1):
Number of leading layers that use a dense MLP; the rest use the MoE block.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: would be nice if we could also move to mlp layer types here but not super important

else nn.Identity()
)

def forward(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The forward is the same now tho, no? Let's remove it and let modular inherit it

@kmswin1

kmswin1 commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor Author

@vasqu Thanks! I’ve addressed these comments as well:

  • Tests: Added tests/models/axk1/ following the youtu test structure. This includes AXK1ModelTester(CausalLMModelTester) with the appropriate MLA and MoE dimensions, an override for the MLA cache shape, and AXK1IntegrationTest with both dynamic- and static-cache generation tests on skt/A.X-K1. The expected completions were verified on our cluster.
  • Docs: Both documentation comments have been addressed.
  • layer_types nit: Agreed that migrating fully to layer_types would be cleaner. For now, I kept first_k_dense_replace because it is inherited from DeepseekV3Config and is present in the released AXK1 configs. I’d be happy to migrate this in a follow-up together with the corresponding DeepseekV3 cleanup.
  • Decoder forward override: Good catch. Instead of retaining the custom decoder forward, I moved post_mlp_layernorm into AXK1MoE, since it is only used on MoE layers. As a result, AXK1DecoderLayer now fully inherits from DeepseekV3DecoderLayer.

Checkpoint compatibility is preserved through a WeightRenaming("post_mlp_layernorm" β†’ "mlp.post_mlp_layernorm") entry in the conversion mapping, following the same pattern used by Laguna. I also verified that generation on the real checkpoint remains unchanged.

@vasqu vasqu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super ready, just a few small last nits on the tests πŸ€—

Comment thread tests/models/axk1/test_modeling_axk1.py Outdated

@require_torch_accelerator
def test_dynamic_cache(self):
EXPECTED_TEXT_COMPLETION = "λŒ€ν•œλ―Όκ΅­μ˜ μˆ˜λ„λŠ” **μ„œμšΈνŠΉλ³„μ‹œ**μž…λ‹ˆλ‹€."

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expectations class please

Comment thread tests/models/axk1/test_modeling_axk1.py Outdated
self.assertEqual(EXPECTED_TEXT_COMPLETION, dynamic_text.strip())

@require_torch_accelerator
def test_static_cache(self):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also add a logits test (minimax m2 style)

@kmswin1

kmswin1 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

@vasqu Thanks! both comments have been addressed:

  • Expectations class: Both the generation expectations and the newly added logits expectations now use Expectations, keyed by ("cuda", 8). The reference values were recorded on an A100.

  • Logits integration test: Added test_model_logits_batched following the MiniMax-M2 pattern. It uses the same left-padded batched dummy input, compares the final 3 Γ— 3 logits slices, and uses assert_close with a tolerance of 1e-3.

The reference logits were generated from the actual checkpoint on our cluster using A100 GPUs in BF16.

@vasqu

vasqu commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

run-slow: axk1

@github-actions

Copy link
Copy Markdown
Contributor

Workflow Run βš™οΈ

This comment contains run-slow, running the specified jobs:

models: ["models/axk1"]
quantizations: []

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@vasqu vasqu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok seems like we have a few smaller issues. Please check https://github.com/huggingface/transformers/actions/runs/30011251731/job/89220223022. You can run all tests with RUN_SLOW=1 pytest tests/models/axk1/test_modeling_axk1.py

Failing tests are

  • test_cpu_offload
  • test_flex_attention_with_grads
  • Integration tests
    • The model type does not match the hub config so this is one issue we need to resolve
    • The model is too big for our CI (24GB VRAM), would it be possible to create a randomized small model

@github-actions

Copy link
Copy Markdown
Contributor

CI Results

Workflow Run βš™οΈ

Commit Info

Context Commit Description
RUN 097ecdc9 workflow commit (merge commit)
PR 8e321a52 branch commit (from PR)
main 6b0f8227 base commit (on main)

⚠️ No test being reported (jobs are skipped or cancelled)!

@kmswin1

kmswin1 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

@vasqu Okay. I'll fix it now.

@vasqu

vasqu commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

@kmswin1 Can you focus on the random model, I will try to check the other tests as well

Comment thread tests/models/axk1/test_modeling_axk1.py Resolved
@vasqu

vasqu commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Tests are fixed, I have a local CI so I can fast track whenever you are ready with the integration tests

Comment thread tests/models/axk1/test_modeling_axk1.py Resolved
@kmswin1
kmswin1 force-pushed the feature/add_axk1 branch from 964c771 to 87384de Compare July 23, 2026 15:12
@vasqu

vasqu commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

run-slow: axk1

@vasqu

vasqu commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Merging after this last run slow ci 🫑 thanks a lot

@github-actions

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, axk1

@github-actions

Copy link
Copy Markdown
Contributor

Workflow Run βš™οΈ

This comment contains run-slow, running the specified jobs:

models: ["models/axk1"]
quantizations: []

@github-actions

Copy link
Copy Markdown
Contributor

CI Results

Workflow Run βš™οΈ

Commit Info

Context Commit Description
RUN 91fb54f3 workflow commit (merge commit)
PR d375f5e7 branch commit (from PR)
main 00d795b4 base commit (on main)

βœ… No failing test specific to this PR πŸŽ‰ πŸ‘ !

@vasqu
vasqu enabled auto-merge July 23, 2026 16:30
@vasqu
vasqu added this pull request to the merge queue Jul 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 30025346200:2
Result: success | Jobs: 16 | Tests: 174,500 | Failures: 0 | Duration: 13h 50m

Merged via the queue into huggingface:main with commit c27860a Jul 23, 2026
214 of 216 checks passed
stevhliu pushed a commit to stevhliu/transformers that referenced this pull request Jul 30, 2026
* add_axk1

* fix TP bug of load balancing bias term

* change default value of n_routed_experts and max_position_embeddings

* address review comments

* change model type to snake case, and rebase

* refactor AXK1 to inherit from deepseek_v3 and modernize config

  - inherit from deepseek_v3 (AXK1Attention / AXK1TopkRouter / AXK1MoE / AXK1Experts /
    AXK1DecoderLayer / AXK1Model / AXK1ForCausalLM), keeping the A.X-K1-specific
    post_mlp_layernorm (applied to the MoE block output) as the only override
  - config: rewrite in @strict style inheriting DeepseekV3Config (class-level typed
    attributes, automatic docstring), moved into modular_axk1.py; drop pretraining_tp /
    moe_layer_freq / the rope-float override; inherit base_model_ep_plan
  - experts AXK1NaiveMoe(MixtralExperts) -> AXK1Experts(DeepseekV3Experts) (same stacked
    layout / keys); routing now lives in the router
  - _keys_to_ignore_on_load_unexpected = ["inv_freq"]
  - add docs (model_doc/axk1.md + toctree)
  - bump copyrights to 2026

  Verified on the light checkpoint: loads with the only newly-initialized key being the
  (zero) e_score_correction_bias the checkpoint omits, and greedy generation is coherent
  ("λŒ€ν•œλ―Όκ΅­μ˜ μˆ˜λ„λŠ”" -> " μ„œμšΈμ΄λ‹€.").

* address second review round: youtu-style tests, docs cleanup, inherit decoder layer

* use Expectations class and add batched logits integration test

* fixup repo

* idk why but go

* Add random small model test

* test fixes

* oops that was from my local

* add integration test

* fixup integration tests

* style

---------

Co-authored-by: minsangkim <kmswin1@sktelecom.com>
Co-authored-by: vasqu <antonprogamer@gmail.com>
Sainava pushed a commit to Sainava/Sai-transformers that referenced this pull request Aug 3, 2026
* add_axk1

* fix TP bug of load balancing bias term

* change default value of n_routed_experts and max_position_embeddings

* address review comments

* change model type to snake case, and rebase

* refactor AXK1 to inherit from deepseek_v3 and modernize config

  - inherit from deepseek_v3 (AXK1Attention / AXK1TopkRouter / AXK1MoE / AXK1Experts /
    AXK1DecoderLayer / AXK1Model / AXK1ForCausalLM), keeping the A.X-K1-specific
    post_mlp_layernorm (applied to the MoE block output) as the only override
  - config: rewrite in @strict style inheriting DeepseekV3Config (class-level typed
    attributes, automatic docstring), moved into modular_axk1.py; drop pretraining_tp /
    moe_layer_freq / the rope-float override; inherit base_model_ep_plan
  - experts AXK1NaiveMoe(MixtralExperts) -> AXK1Experts(DeepseekV3Experts) (same stacked
    layout / keys); routing now lives in the router
  - _keys_to_ignore_on_load_unexpected = ["inv_freq"]
  - add docs (model_doc/axk1.md + toctree)
  - bump copyrights to 2026

  Verified on the light checkpoint: loads with the only newly-initialized key being the
  (zero) e_score_correction_bias the checkpoint omits, and greedy generation is coherent
  ("λŒ€ν•œλ―Όκ΅­μ˜ μˆ˜λ„λŠ”" -> " μ„œμšΈμ΄λ‹€.").

* address second review round: youtu-style tests, docs cleanup, inherit decoder layer

* use Expectations class and add batched logits integration test

* fixup repo

* idk why but go

* Add random small model test

* test fixes

* oops that was from my local

* add integration test

* fixup integration tests

* style

---------

Co-authored-by: minsangkim <kmswin1@sktelecom.com>
Co-authored-by: vasqu <antonprogamer@gmail.com>
@ydshieh ydshieh mentioned this pull request Aug 14, 2026
1 task done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants