Qwen 3.5

Qwen3.5 is a hybrid architecture model series combining Gated DeltaNet linear attention with standard Transformer attention. All Qwen3.5 models are early-fusion vision-language models: dense variants use Qwen3_5ForConditionalGeneration and MoE variants use Qwen3_5MoeForConditionalGeneration.

Getting started

  1. Install Axolotl following the installation guide.

  2. Install Cut Cross Entropy to reduce training VRAM usage.

  3. Install FLA for sample packing support with the Gated DeltaNet linear attention layers:

uv pip uninstall causal-conv1d && uv pip install flash-linear-attention==0.5.2

FLA is required when sample_packing: true. Without it, training raises a RuntimeError on packed sequences. Vision configs use sample_packing: false so FLA is optional there.

  1. Pick any config from the table below and run:

    axolotl train examples/qwen3.5/<config>.yaml

Available configs:

Config Model Type Peak VRAM
9b-lora-vision.yaml Qwen3.5-9B Vision+text LoRA, single GPU —
9b-fft-vision.yaml Qwen3.5-9B Vision+text FFT, single GPU ~61 GiB
27b-qlora.yaml Qwen3.5-27B Dense, text-only QLoRA ~47 GiB
27b-fft.yaml Qwen3.5-27B Dense, text-only FFT (vision frozen) ~53 GiB
27b-qlora-fsdp.yaml Qwen3.5-27B Dense, text-only QLoRA + FSDP2 —
35b-a3b-moe-qlora.yaml Qwen3.5-35B-A3B MoE, text-only QLoRA —
35b-a3b-moe-qlora-fsdp.yaml Qwen3.5-35B-A3B MoE, text-only QLoRA + FSDP2 —
122b-a10b-moe-qlora.yaml Qwen3.5-122B-A10B MoE, text-only QLoRA —
122b-a10b-moe-qlora-fsdp.yaml Qwen3.5-122B-A10B MoE, text-only QLoRA + FSDP2 —
qwen3.6-35b-a3b-moe-qlora.yaml Qwen3.6-35B-A3B MoE, text-only QLoRA (see Qwen3.6 and Edge0) —

Gated DeltaNet Linear Attention

Qwen3.5 interleaves standard attention with Gated DeltaNet linear attention layers. To apply LoRA to them, add to lora_target_modules:

lora_target_modules:
  # ... standard projections ...
  - linear_attn.in_proj_qkv
  - linear_attn.in_proj_z
  - linear_attn.out_proj

Routed Experts (MoE)

To apply LoRA to routed expert parameters, add lora_target_parameters:

lora_target_parameters:
  - mlp.experts.gate_up_proj
  - mlp.experts.down_proj
#  - mlp.gate.weight  # router

Shared Experts (MoE)

Shared experts use nn.Linear (unlike routed experts which are 3D nn.Parameter tensors), so they can be targeted via lora_target_modules. To also train shared expert projections alongside attention, uncomment gate_proj, up_proj and down_proj in lora_target_modules:

lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  # - gate_proj
  # - up_proj
  # - down_proj

Use lora_target_parameters (see Routed Experts above) to target routed experts separately.

Qwen3.6 and Edge0

Qwen3.6-35B-A3B keeps the qwen3_5_moe architecture and trains with qwen3.6-35b-a3b-moe-qlora.yaml. Its extra mtp.* (multi-token prediction) tensors are dropped on load.

Edge0 checkpoints are MLX int4 exports (Edge0-35B-A3B-preview of Qwen3.6-35B-A3B, Edge0-8B-A1B-preview of Ling-3.0-tiny) and cannot be loaded by transformers. Fine-tune the base model instead and quantize afterwards.

Nex-N2.5-mini shares this architecture with its own chat template.

TIPS

  • For inference hyp, please see the respective model card details.
  • You can run a full finetuning of smaller configs by removing adapter: qlora and load_in_4bit: true. See Multi-GPU below.
  • Read more on loading your own dataset at docs.
  • The dataset format follows the OpenAI Messages format as seen here.
  • For multimodal finetuning, set processor_type: AutoProcessor, skip_prepare_dataset: true, and remove_unused_columns: false as shown in 9b-lora-vision.yaml.

Optimization Guides