GLM-4.7-Flash

GLM-4.7-Flash is a 30B-A3B MoE model by Z.ai.

This guide shows how to fine-tune it with Axolotl.

Getting started

  1. Install Axolotl following the installation guide.

  2. Install Cut Cross Entropy to reduce training VRAM usage.

  3. Run the finetuning example:

# QLoRA
# - no target experts (1x48GB @ ~24GiB/GPU)
# - target experts (1x48GB @ ~34GiB/GPU)
axolotl train examples/glm47-flash/qlora.yaml

# QLoRA FSDP2 no target experts (2x48GB @ ~29GiB/GPU)
axolotl train examples/glm47-flash/qlora_fsdp.yaml
# LoRA
# - no target experts (1x48GB @ ~35GiB/GPU)
# - target experts (1x48GB @ OOM. Projected ~45-50GiB/GPU)
axolotl train examples/glm47-flash/lora.yaml

# LoRA FSDP2 no target experts (2x48GB @ ~43GiB/GPU)
axolotl train examples/glm47-flash/lora_fsdp.yaml

MoE Expert Quantization & Expert LoRA

This model quantize expert weights on load. To learn about expert quantization, expert LoRA targeting, and related limitations, see the MoE Expert Quantization docs.

Limitations

  • lora_target_linear: Incompatible for this model.
  • LoRA kernels: The q/k/v/o kernels don’t apply to this model’s MLA attention, and enabling lora_qkv_kernel/lora_o_kernel raises an error. The example configs set lora_*_kernel: false.
  • LoRA targets: The attention projections are q_a_proj, q_b_proj, kv_a_proj_with_mqa, kv_b_proj and o_proj. The usual q_proj/k_proj/v_proj don’t exist here, so targeting them silently adapts only o_proj.

TIPS

  • For inference, the official Z.ai team recommends these default settings (most tasks):
    • temperature: 1.0
    • top_p: 0.95
    • max_new_tokens: 131072
  • You can run a full finetuning by removing adapter: qlora, load_in_4bit: true, and quantize_moe_experts: true from the config. This is heavy, so we have not tested this.
  • Read more on how to load your own dataset at docs.

Optimization Guides

Please check the Optimizations doc.