datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
2l-bilinear-attn
pile
Quadratic/bilinear attention causal language model trained with the tensor-mars research stack. This repository packages the final checkpoint, configuration, and reference model code.
## Training configuration
```yaml
batch_size: 384
max_steps: 33333
warmup_steps: 200
lr: 0.0003
optimizer: Muon + AdamW
dtype: bfloat16
grad_clip: 1.0
```
## Data + tokenizer
- Context length: 512 | Vocab size: 4096
## Metrics
- **train_loss**: 3.9820… See the full description on the dataset page: https://huggingface.co/datasets/melephant/2l-bilinear-attn.hunyuan-image3-dit-attn-capture
HunyuanImage-3.0 DiT attention capture — handoff
Real attention inputs (q/k/v + mask) captured from the HunyuanImage-3.0 DiT, for
kernel work: choosing/implementing an attention backend that can serve this
model. The mask here is structural, not a padding mask, so it is the part
that constrains what a kernel can accept.
What's in this bundle
File
Size
Contents
out/call0_rank0.npz
16.5 MiB
prefill — q/k/v + mask
out/call32_rank0.npz
14.0 MiB
denoise —… See the full description on the dataset page: https://huggingface.co/datasets/Yi30/hunyuan-image3-dit-attn-capture.blip-kd-student-attn-distill-fashion200k-15k2l-bilinear-attn-v2