sukhdeveyash/XLS-R-SLS-Deepfake-Detection

XLS-R + SLS Classifier for Audio Deepfake Detection

Reproduction of "Audio Deepfake Detection with XLS-R and SLS Classifier" (Zhang et al., ACM Multimedia 2024).

The Selective Layer Summarization (SLS) classifier extracts attention-weighted features from all 24 transformer layers of XLS-R 300M (wav2vec 2.0), then classifies bonafide vs. spoofed speech via a lightweight fully-connected head. RawBoost (algo=3, SSI) data augmentation is applied during training.

Available Checkpoints

Recommended: Use v1/epoch_2.pth — it generalizes better to unseen attack types (DF, In-the-Wild).

Original authors' pretrained models

The original pretrained checkpoints from Zhang et al. are available from:

• —Google Drive
• —Baidu Pan (password: shan)

Results

v1 closely reproduces the paper results. v2 improves LA slightly but degrades DF and In-the-Wild due to overfitting to the LA validation domain — a well-documented cross-domain generalization problem in audio deepfake detection (Muller et al., Interspeech 2022).

Training Configuration

Both experiments share the following setup:

v1 specifics

• —Early stopping: patience=1 on training loss
• —No validation set
• —4 epochs trained, best at epoch 2 (train loss = 0.000661)

v2 specifics

• —Early stopping: patience=10 on validation loss
• —Validation: ASVspoof2019 LA dev (24,844 trials)
• —27 epochs trained, best at epoch 16 (valloss = 0.000468, valacc = 99.99%)
• —Bug fixes: torch.no_grad() in validation loop, correct best_val_loss tracking

Usage

Download checkpoint

from huggingface_hub import hf_hub_download

# Download v1 checkpoint (recommended)
checkpoint_path = hf_hub_download(
    repo_id="sukhdeveyash/XLS-R-SLS-Deepfake-Detection",
    filename="v1/epoch_2.pth"
)

# Download v2 checkpoint
# checkpoint_path = hf_hub_download(
#     repo_id="sukhdeveyash/XLS-R-SLS-Deepfake-Detection",
#     filename="v2/epoch_16.pth"
# )

Load and run inference

import torch
from model import Model  # from the GitHub repo

device = "cuda" if torch.cuda.is_available() else "cpu"

model = Model(device=device, ssl_cpkt_path="xlsr2_300m.pt")
model.load_state_dict(torch.load(checkpoint_path, map_location=device))
model = model.to(device)
model.eval()

Full training and evaluation code: GitHub Repository

Requirements

• —Python 3.7+
• —PyTorch 1.13.1 (CUDA 11.7)
• —fairseq (commit a54021305d6b3c)
• —XLS-R 300M base checkpoint (xlsr2_300m.pt) from fairseq

See environment.yml in the GitHub repo for the full environment.

Citation

@inproceedings{zhang2024audio,
  title={Audio Deepfake Detection with XLS-R and SLS Classifier},
  author={Zhang, Qishan and Wen, Shuangbing and Hu, Tao},
  booktitle={Proceedings of the 32nd ACM International Conference on Multimedia},
  year={2024},
  publisher={ACM}
}

Acknowledgements

• —XLS-R (Babu et al., 2022)
• —RawBoost (Tak et al., Odyssey 2022)
• —ASVspoof Challenge