Repository navigation
add_axk1 - #46867
add_axk1#46867
Conversation
|
Thanks @kmswin1, we're taking a look |
acb99b0 to
89848e1
Compare
Cyrilvallez
left a comment
There was a problem hiding this comment.
Hey! Sorry for the delay! Very nice first implem in general!
We need to make sure that we follow naming conventions, i.e. snake_case for model_type, filenames etc, and CamelCase for classes though, no uppercases.
Missing tests as well!
| ("autoformer", "AutoformerConfig"), | ||
| ("AXK1", "AXK1Config"), |
There was a problem hiding this comment.
We are using snake_case and CamelCase conventions in Transformers!
There was a problem hiding this comment.
Thank you for your review!
Our model name is A.X-K Series (e.g. not A.x-k1), and our standard convention is to use uppercase. Could we maintain this naming rule?
There was a problem hiding this comment.
If possible, we would prefer to keep the uppercase naming to stay consistent with the official A.X-K Series naming. That said, I understand the library convention and donβt want this to block the PR. If keeping the uppercase form is not acceptable, Iβm happy to update the internal model_type, module names, and registry keys to lowercase/snake_case, e.g. axk1, while keeping the public class names as AXK1Config / AXK1Model.
There was a problem hiding this comment.
Alright, let's do that then!
| ("sam3_lite_text", "Sam3LiteTextConfig"), | ||
| ("sam3_lite_text_text_model", "Sam3LiteTextTextConfig"), | ||
| ("sam3_tracker", "Sam3TrackerConfig"), |
There was a problem hiding this comment.
Leftover from an unrelated local branch removed. The auto mappings now only add the AXK1 entries.
| ("sam3_lite_text", "SAM3-LiteText"), | ||
| ("sam3_lite_text_text_model", "SAM3-LiteText Text Model"), |
There was a problem hiding this comment.
Leftover from an unrelated local branch removed. The auto mappings now only add the AXK1 entries.
There was a problem hiding this comment.
Leftover from an unrelated local branch removed. The auto mappings now only add the AXK1 entries.
| ) | ||
|
|
||
| # This is tied to the processing `-` -> `_` in `model_type_to_module_name`. For example, instead of putting | ||
| # `transfo-xl` (as in `CONFIG_MAPPING_NAMES`), we should use `transfo_xl`. | ||
| DEPRECATED_MODELS = [] | ||
|
|
||
| SPECIAL_MODEL_TYPE_TO_MODULE_NAME = OrderedDict[str, str]( | ||
| [ | ||
| ("AXK1", "AXK1"), |
There was a problem hiding this comment.
As I said, we really want to fllow snake_case and CamelCase conventions
There was a problem hiding this comment.
Acknowledged. we'd like to keep the AXK1 casing for now since the released checkpoints on the Hub use model_type: "AXK1" and renaming would break AutoConfig for existing configs. Happy to migrate to axk1/Axk1* in a follow-up if you feel strongly; let us know your preference for handling the existing checkpoints.
There was a problem hiding this comment.
Yes, the model_type itself should really be in snake_case - it's very easy to change the config directly on the hub, just a one-line change!
|
|
||
| AXK1_PRETRAINED_CONFIG_ARCHIVE_MAP = {} |
|
|
||
| class AXK1NaiveMoe(MixtralExperts): |
There was a problem hiding this comment.
| class AXK1NaiveMoe(MixtralExperts): | |
| class Axk1Experts(MixtralExperts): |
There was a problem hiding this comment.
This one is tied to the casing discussion above.
| def route_tokens_to_experts(self, router_logits): | ||
| router_logits = router_logits.sigmoid() | ||
| if self.gate.e_score_correction_bias is not None: | ||
| router_logits_for_choice = router_logits + self.gate.e_score_correction_bias | ||
| else: | ||
| router_logits_for_choice = router_logits | ||
| group_scores = ( | ||
| router_logits_for_choice.view(-1, self.n_group, self.n_routed_experts // self.n_group) | ||
| .topk(2, dim=-1)[0] | ||
| .sum(dim=-1) | ||
| ) | ||
| group_idx = torch.topk(group_scores, k=self.topk_group, dim=-1, sorted=False)[1] | ||
| group_mask = torch.zeros_like(group_scores) | ||
| group_mask.scatter_(1, group_idx, 1) | ||
| score_mask = ( | ||
| group_mask.unsqueeze(-1) | ||
| .expand(-1, self.n_group, self.n_routed_experts // self.n_group) | ||
| .reshape(-1, self.n_routed_experts) | ||
| ) | ||
| scores_for_choice = router_logits_for_choice.masked_fill(~score_mask.bool(), 0.0) | ||
| topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1] | ||
| topk_weights = router_logits.gather(1, topk_indices) | ||
| if self.norm_topk_prob: | ||
| denominator = topk_weights.sum(dim=-1, keepdim=True) + 1e-20 | ||
| topk_weights /= denominator | ||
| topk_weights = topk_weights * self.routed_scaling_factor | ||
| return topk_indices, topk_weights |
There was a problem hiding this comment.
This should be part of router's forward!
There was a problem hiding this comment.
Done. Routing (sigmoid, group top-k, normalization, scaling) now lives in AXK1TopkRouter.forward, which returns (topk_indices, topk_weights); AXK1MoE is a thin wrapper.
| if self.config.rope_interleave: | ||
| q_rot, k_rot = apply_rotary_pos_emb_interleave(q_rot, k_rot, cos, sin) | ||
| else: | ||
| q_rot, k_rot = apply_rotary_pos_emb(q_rot, k_rot, cos, sin) |
There was a problem hiding this comment.
Do we ever have both paths in released checkpoints?
There was a problem hiding this comment.
No. all released checkpoints use the interleaved layout, so the flag and the non-interleaved branch are removed; the model always applies interleaved RoPE.
| self.input_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps) | ||
| self.post_attention_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps) |
There was a problem hiding this comment.
| self.input_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps) | |
| self.post_attention_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps) | |
| self.input_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps) | |
| self.post_attention_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps) is not self.is_moe_layer else nn.Identity() |
There was a problem hiding this comment.
Done: dense layers get nn.Identity() and the branch in the decoder forward is gone.
| def forward( | ||
| self, | ||
| hidden_states: torch.Tensor, | ||
| attention_mask: torch.Tensor | None = None, | ||
| position_ids: torch.LongTensor | None = None, | ||
| past_key_values: Cache | None = None, | ||
| use_cache: bool | None = False, | ||
| position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None, | ||
| **kwargs: Unpack[TransformersKwargs], | ||
| ) -> torch.Tensor: | ||
| residual = hidden_states | ||
| hidden_states = self.input_layernorm(hidden_states) | ||
|
|
||
| hidden_states, _ = self.self_attn( | ||
| hidden_states=hidden_states, | ||
| attention_mask=attention_mask, | ||
| position_ids=position_ids, | ||
| past_key_values=past_key_values, | ||
| use_cache=use_cache, | ||
| position_embeddings=position_embeddings, | ||
| **kwargs, | ||
| ) | ||
| hidden_states = residual + hidden_states | ||
|
|
||
| residual = hidden_states | ||
| hidden_states = self.post_attention_layernorm(hidden_states) | ||
| hidden_states = self.mlp(hidden_states) | ||
| if self.is_moe_layer: | ||
| hidden_states = self.post_mlp_layernorm(hidden_states) | ||
| hidden_states = residual + hidden_states | ||
| return hidden_states |
There was a problem hiding this comment.
To remove, with the previous comment about Identity
There was a problem hiding this comment.
Done: the branch is removed; post_mlp_layernorm is now applied unconditionally (nn.Identity on dense layers).
Thank you for the review! Iβll go through the comments one by one and update the PR accordingly. |
|
Thanks again for the thorough review! I addressed the comments in the latest commit:
I also verified that this refactor preserves behavior: generation with the released checkpoint is unchanged after the update. The only remaining open point is the |
|
Hey @kmswin1! Alright, let's go with the tradeoff of model_type in snake_case, while keeping ModuleClasses as AXK1Module then! |
|
Ping me once all this is done for a new review! |
# Conflicts: # src/transformers/models/auto/configuration_auto.py # src/transformers/utils/auto_docstring.py
|
Thanks again! @Cyrilvallez Iβve updated the PR accordingly:
Please let me know if there is anything else youβd like me to adjust. |
vasqu
left a comment
There was a problem hiding this comment.
Some comments from my side π€
| if getattr(config, "topk_method", "noaux_tc") == "noaux_tc": | ||
| self.register_buffer("e_score_correction_bias", torch.zeros(self.n_routed_experts)) | ||
| else: | ||
| self.e_score_correction_bias = None |
There was a problem hiding this comment.
Are there really models that use this option? In any case we can still inherit from modular
- The init can be used with super and only the e score needs to change
- For the forward, we rewrite but add a small comment for the exact change
| self.intermediate_dim = config.moe_intermediate_size | ||
|
|
||
|
|
||
| class AXK1MoE(nn.Module): |
There was a problem hiding this comment.
Pretty sure that e.g. deepseek v2 has a similar strucute but you need to sync with main
| return hidden_states | ||
|
|
||
|
|
||
| class AXK1Attention(nn.Module): |
There was a problem hiding this comment.
This looks very deepseek v3 ish? Let's use modular please
| nn.Module.__init__(self) | ||
| self.config = config | ||
| self.layer_idx = layer_idx | ||
| self.hidden_size = config.hidden_size | ||
|
|
||
| self.self_attn = AXK1Attention(config=config, layer_idx=layer_idx) | ||
|
|
||
| self.is_moe_layer = ( | ||
| config.n_routed_experts is not None | ||
| and layer_idx >= config.first_k_dense_replace | ||
| and layer_idx % config.moe_layer_freq == 0 | ||
| ) | ||
| if self.is_moe_layer: | ||
| self.mlp = AXK1MoE(config) | ||
| else: | ||
| self.mlp = AXK1MLP(config) | ||
|
|
||
| self.input_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps) | ||
| self.post_attention_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps) | ||
| # `post_mlp_layernorm` only exists on MoE layers. | ||
| if self.is_moe_layer: | ||
| self.post_mlp_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps) | ||
| else: | ||
| self.post_mlp_layernorm = nn.Identity() |
There was a problem hiding this comment.
| nn.Module.__init__(self) | |
| self.config = config | |
| self.layer_idx = layer_idx | |
| self.hidden_size = config.hidden_size | |
| self.self_attn = AXK1Attention(config=config, layer_idx=layer_idx) | |
| self.is_moe_layer = ( | |
| config.n_routed_experts is not None | |
| and layer_idx >= config.first_k_dense_replace | |
| and layer_idx % config.moe_layer_freq == 0 | |
| ) | |
| if self.is_moe_layer: | |
| self.mlp = AXK1MoE(config) | |
| else: | |
| self.mlp = AXK1MLP(config) | |
| self.input_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps) | |
| self.post_attention_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps) | |
| # `post_mlp_layernorm` only exists on MoE layers. | |
| if self.is_moe_layer: | |
| self.post_mlp_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps) | |
| else: | |
| self.post_mlp_layernorm = nn.Identity() | |
| super().__init__(config, layer_idx) | |
| self.mlp = AXK1MoE(config) if config.mlp_layer_types[layer_idx] == "sparse" else AXK1MLP(config) | |
| self.post_attention_layernorm = AXK1RMSNorm(config.hidden_size, eps=config.rms_norm_eps) if config.mlp_layer_types[layer_idx] == "sparse" else nn.Identity() |
let's use mlp layer types instead please
|
|
||
| class AXK1PreTrainedModel(LlamaPreTrainedModel): | ||
| _keep_in_fp32_modules_strict = ["e_score_correction_bias"] | ||
| _keys_to_ignore_on_load_unexpected = [r"model\.layers\.\d+\.self_attn\.rotary_emb\.inv_freq"] |
There was a problem hiding this comment.
| _keys_to_ignore_on_load_unexpected = [r"model\.layers\.\d+\.self_attn\.rotary_emb\.inv_freq"] | |
| _keys_to_ignore_on_load_unexpected = [r"inv_freq"] |
- inherit from deepseek_v3 (AXK1Attention / AXK1TopkRouter / AXK1MoE / AXK1Experts /
AXK1DecoderLayer / AXK1Model / AXK1ForCausalLM), keeping the A.X-K1-specific
post_mlp_layernorm (applied to the MoE block output) as the only override
- config: rewrite in @strict style inheriting DeepseekV3Config (class-level typed
attributes, automatic docstring), moved into modular_axk1.py; drop pretraining_tp /
moe_layer_freq / the rope-float override; inherit base_model_ep_plan
- experts AXK1NaiveMoe(MixtralExperts) -> AXK1Experts(DeepseekV3Experts) (same stacked
layout / keys); routing now lives in the router
- _keys_to_ignore_on_load_unexpected = ["inv_freq"]
- add docs (model_doc/axk1.md + toctree)
- bump copyrights to 2026
Verified on the light checkpoint: loads with the only newly-initialized key being the
(zero) e_score_correction_bias the checkpoint omits, and greedy generation is coherent
("λνλ―Όκ΅μ μλλ" -> " μμΈμ΄λ€.").
|
I have also applied the relevant changes discussed during the A.X-K2 review to A.X-K1, including the updated modular/config patterns, naming conventions, documentation, tests, and repository alignment where applicable. |
vasqu
left a comment
There was a problem hiding this comment.
Small comments left. One important thing tho: We need tests, e.g. following deepseek v3. I think youtu is the best example because it uses our new structure for tests: https://github.com/huggingface/transformers/blob/main/tests/models/youtu/test_modeling_youtu.py
| dtype=torch.bfloat16, | ||
| device_map="auto", |
There was a problem hiding this comment.
| dtype=torch.bfloat16, | |
| device_map="auto", |
| tokenizer = AutoTokenizer.from_pretrained("skt/A.X-K1") | ||
| model = AutoModelForCausalLM.from_pretrained( | ||
| "skt/A.X-K1", | ||
| dtype=torch.bfloat16, |
There was a problem hiding this comment.
| dtype=torch.bfloat16, |
| first_k_dense_replace (`int`, *optional*, defaults to 1): | ||
| Number of leading layers that use a dense MLP; the rest use the MoE block. |
There was a problem hiding this comment.
Nit: would be nice if we could also move to mlp layer types here but not super important
| else nn.Identity() | ||
| ) | ||
|
|
||
| def forward( |
There was a problem hiding this comment.
The forward is the same now tho, no? Let's remove it and let modular inherit it
|
@vasqu Thanks! Iβve addressed these comments as well:
Checkpoint compatibility is preserved through a |
vasqu
left a comment
There was a problem hiding this comment.
Super ready, just a few small last nits on the tests π€
|
|
||
| @require_torch_accelerator | ||
| def test_dynamic_cache(self): | ||
| EXPECTED_TEXT_COMPLETION = "λνλ―Όκ΅μ μλλ **μμΈνΉλ³μ**μ λλ€." |
| self.assertEqual(EXPECTED_TEXT_COMPLETION, dynamic_text.strip()) | ||
|
|
||
| @require_torch_accelerator | ||
| def test_static_cache(self): |
There was a problem hiding this comment.
Can we also add a logits test (minimax m2 style)
|
@vasqu Thanks! both comments have been addressed:
The reference logits were generated from the actual checkpoint on our cluster using A100 GPUs in BF16. |
|
run-slow: axk1 |
|
This comment contains models: ["models/axk1"] |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
vasqu
left a comment
There was a problem hiding this comment.
Ok seems like we have a few smaller issues. Please check https://github.com/huggingface/transformers/actions/runs/30011251731/job/89220223022. You can run all tests with RUN_SLOW=1 pytest tests/models/axk1/test_modeling_axk1.py
Failing tests are
- test_cpu_offload
- test_flex_attention_with_grads
- Integration tests
- The model type does not match the hub config so this is one issue we need to resolve
- The model is too big for our CI (24GB VRAM), would it be possible to create a randomized small model
|
@vasqu Okay. I'll fix it now. |
|
@kmswin1 Can you focus on the random model, I will try to check the other tests as well |
|
Tests are fixed, I have a local CI so I can fast track whenever you are ready with the integration tests |
964c771 to
87384de
Compare
|
run-slow: axk1 |
|
Merging after this last run slow ci π«‘ thanks a lot |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, axk1 |
|
This comment contains models: ["models/axk1"] |
CI recapDashboard: View test results in Grafana |
c27860a
* add_axk1
* fix TP bug of load balancing bias term
* change default value of n_routed_experts and max_position_embeddings
* address review comments
* change model type to snake case, and rebase
* refactor AXK1 to inherit from deepseek_v3 and modernize config
- inherit from deepseek_v3 (AXK1Attention / AXK1TopkRouter / AXK1MoE / AXK1Experts /
AXK1DecoderLayer / AXK1Model / AXK1ForCausalLM), keeping the A.X-K1-specific
post_mlp_layernorm (applied to the MoE block output) as the only override
- config: rewrite in @strict style inheriting DeepseekV3Config (class-level typed
attributes, automatic docstring), moved into modular_axk1.py; drop pretraining_tp /
moe_layer_freq / the rope-float override; inherit base_model_ep_plan
- experts AXK1NaiveMoe(MixtralExperts) -> AXK1Experts(DeepseekV3Experts) (same stacked
layout / keys); routing now lives in the router
- _keys_to_ignore_on_load_unexpected = ["inv_freq"]
- add docs (model_doc/axk1.md + toctree)
- bump copyrights to 2026
Verified on the light checkpoint: loads with the only newly-initialized key being the
(zero) e_score_correction_bias the checkpoint omits, and greedy generation is coherent
("λνλ―Όκ΅μ μλλ" -> " μμΈμ΄λ€.").
* address second review round: youtu-style tests, docs cleanup, inherit decoder layer
* use Expectations class and add batched logits integration test
* fixup repo
* idk why but go
* Add random small model test
* test fixes
* oops that was from my local
* add integration test
* fixup integration tests
* style
---------
Co-authored-by: minsangkim <kmswin1@sktelecom.com>
Co-authored-by: vasqu <antonprogamer@gmail.com>
* add_axk1
* fix TP bug of load balancing bias term
* change default value of n_routed_experts and max_position_embeddings
* address review comments
* change model type to snake case, and rebase
* refactor AXK1 to inherit from deepseek_v3 and modernize config
- inherit from deepseek_v3 (AXK1Attention / AXK1TopkRouter / AXK1MoE / AXK1Experts /
AXK1DecoderLayer / AXK1Model / AXK1ForCausalLM), keeping the A.X-K1-specific
post_mlp_layernorm (applied to the MoE block output) as the only override
- config: rewrite in @strict style inheriting DeepseekV3Config (class-level typed
attributes, automatic docstring), moved into modular_axk1.py; drop pretraining_tp /
moe_layer_freq / the rope-float override; inherit base_model_ep_plan
- experts AXK1NaiveMoe(MixtralExperts) -> AXK1Experts(DeepseekV3Experts) (same stacked
layout / keys); routing now lives in the router
- _keys_to_ignore_on_load_unexpected = ["inv_freq"]
- add docs (model_doc/axk1.md + toctree)
- bump copyrights to 2026
Verified on the light checkpoint: loads with the only newly-initialized key being the
(zero) e_score_correction_bias the checkpoint omits, and greedy generation is coherent
("λνλ―Όκ΅μ μλλ" -> " μμΈμ΄λ€.").
* address second review round: youtu-style tests, docs cleanup, inherit decoder layer
* use Expectations class and add batched logits integration test
* fixup repo
* idk why but go
* Add random small model test
* test fixes
* oops that was from my local
* add integration test
* fixup integration tests
* style
---------
Co-authored-by: minsangkim <kmswin1@sktelecom.com>
Co-authored-by: vasqu <antonprogamer@gmail.com>
Add A.X K1 model architecture
What does this PR do?
This PR adds support for A.X K1, a large-scale Mixture-of-Experts (MoE) language model developed by SK Telecom. A.X K1 contains 519B total parameters with 33B active parameters per token, enabling strong reasoning performance with practical inference efficiency.
Paper: A.X K1 Technical Report (arXiv:2601.09200)
Model: skt/A.X-K1
License: Apache 2.0
Architecture overview
A.X K1 is a decoder-only Transformer with the following key design choices:
Sparse MoE: 61 layers (1 dense + 60 MoE), with 192 routed experts + 1 shared expert per MoE layer. 8 experts + 1 shared expert are activated per token.
Multi-Latent Attention (MLA): Compressed KV representations for efficient long-context inference.
Post-MLP RMSNorm: An additional RMSNorm applied after the MLP/MoE block in each layer, improving training stability in large-scale sparse settings.
Multi-Token Prediction (MTP): An auxiliary training objective that also enables speculative decoding at inference time.
Hybrid Reasoning (Think / Non-Think): Supports user-controllable reasoning depth within a single unified model.
Context length: 131,072 tokens
Vocab size: 163,840 (BBPE-based tokenizer optimized for EN, KO, ZH, JA, ES and code)
This PR includes
Model implementation (AXK1ForCausalLM, AXK1Model, AXK1Config)
Tokenizer integration (BBPE-based, 163,840 vocab)
Documentation and model card
Tests
Before submitting
I confirm that this is not a pure code agent PR.
This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline, Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
Did you write any new necessary tests?
Who can review?
@ArthurZucker @Cyrilvallez β text model architecture addition