Modelpublic

dronefreak/co-deformable-detr-r50-1x-coco

sourceHugging Faceunknownupdated 17d agoView on Hugging Face

0likes65downloads

Model Card

Co-Deformable-DETR (ResNet-50, 1× / 12 epochs, COCO)

<video controls autoplay loop muted playsinline width="900" poster="https://huggingface.co/dronefreak/co-deformable-detr-r50-1x-coco/resolve/main/assets/demobannerposter.jpg" src="https://huggingface.co/dronefreak/co-deformable-detr-r50-1x-coco/resolve/main/assets/demobanner.mp4"><img src="https://huggingface.co/dronefreak/co-deformable-detr-r50-1x-coco/resolve/main/assets/demobanner_poster.jpg" alt="Co-Deformable-DETR R50 detections on two dashcam clips" width="900"></video>

Disclaimer

This is not an official release. All credit for the method, the architecture, and the trained weights belongs entirely to the original authors: Zhuofan Zong, Guanglu Song, and Yu Liu (SenseTime X-Lab). I claim no contribution to the underlying research or training. I host the same checkpoint the authors published to Google Drive, with a model card and a copy of its config file.

Mirror rationale

The original non-ViT Co-DETR checkpoints are distributed through a single shared Google Drive folder, which is inconvenient for automated and reproducible downloads (no per-file addressing, checksums, or versioning). The authors published their ViT-L checkpoints to the Hub but not the ResNet-50 / Swin ones. I mirror the R50 checkpoint to provide stable artifact hosting, an explicit file checksum (see Provenance), and versioned Hub access.

I mirror only the pretrained weights; I do not redistribute any training or evaluation data. I will transfer or remove this mirror at the original authors' request. Please cite the original paper (see Citation) and use the official repository or the maintained fork `dronefreak/Co-DETR` for code.

Quickstart

⚠️ This is not a plain-PyTorch / transformers model. Inference needs the Co-DETR project code and the OpenMMLab 1.x stack (MMDetection 2.25.3, MMCV-full 1.5.0, PyTorch 1.11). The maintained fork ships a one-command environment setup for exactly this.

bash

# 1. Get the code + a validated environment
git clone https://github.com/dronefreak/Co-DETR && cd Co-DETR
bash tools/setup_codetr_env.sh          # builds the `codetr` conda env
conda activate codetr

# 2. Pull this checkpoint + its config from the Hub
pip install -U "huggingface_hub[cli]"
hf download dronefreak/co-deformable-detr-r50-1x-coco \
    co_deformable_detr_r50_1x_coco.pth co_deformable_detr_r50_1x_coco.py \
    --local-dir checkpoints/

# 3. Run detection on an image / folder / video / webcam
python tools/inference.py \
    --config checkpoints/co_deformable_detr_r50_1x_coco.py \
    --checkpoint checkpoints/co_deformable_detr_r50_1x_coco.pth \
    --input demo/demo.jpg --out-dir outputs/ --save-json

(The bundled config is identical to projects/configs/co_deformable_detr/co_deformable_detr_r50_1x_coco.py in the repo, so --config projects/configs/.../co_deformable_detr_r50_1x_coco.py works too.)

Programmatic loading, if you want the raw state_dict:

python

from huggingface_hub import hf_hub_download
from mmdet.utils import torch_load_compat  # torch.load shim: works on torch 1.11+

repo = "dronefreak/co-deformable-detr-r50-1x-coco"
ckpt_path = hf_hub_download(repo, "co_deformable_detr_r50_1x_coco.pth")
ckpt = torch_load_compat(ckpt_path, map_location="cpu")   # do NOT use raw torch.load here

print(list(ckpt))                       # -> ['state_dict']  (no 'meta', no 'optimizer')
sd = ckpt["state_dict"]
print(len(sd), "tensors")               # -> 676
print(sorted({k.split(".")[0] for k in sd}))
# -> ['backbone', 'bbox_head', 'neck', 'query_head', 'roi_head', 'rpn_head']

To build the full detector object, use mmdet.apis.init_detector(config, checkpoint) (as tools/inference.py does), not load_state_dict directly: the config wires up the backbone, neck, and head registry.

What is Co-DETR / this checkpoint

Co-DETR ("DETRs with Collaborative Hybrid Assignments Training") is a training scheme, not a new architecture. It attaches auxiliary heads that use one-to-many label assignment (an ATSS head and a Faster-R-CNN-style RoI head) alongside the DETR decoder's one-to-one Hungarian matching, and feeds the positive proposals those heads produce back into the decoder as extra queries. This makes the encoder's features more discriminative and speeds up convergence. The auxiliary heads are used only during training: this checkpoint's config sets eval_module='detr', so at inference only the Deformable-DETR query head runs and the RPN / RoI / ATSS heads add no inference-time computation to the deployed detector.

This checkpoint applies that scheme to Deformable-DETR (two-stage, with iterative box refinement) and a ResNet-50 backbone:


Detector	`CoDETR` meta-model; deployed head is a two-stage Deformable-DETR (`CoDeformDETRHead`)
Backbone	ResNet-50, ImageNet-pretrained (`torchvision://resnet50`), stage-1 + all BN frozen
Neck	`ChannelMapper`, 3→4 feature levels, 256 channels, GroupNorm
Queries	300 object queries; `as_two_stage=True`, `mixed_selection=True`, `look_forward_twice=True`
Transformer	6-layer deformable-attention encoder + 6-layer decoder; `num_co_heads=2` collaboration branches (training only)
Aux heads in state-dict	`RPNHead` + `CoStandardRoIHead`, `CoATSSHead`: present in the weights but not executed at inference (`eval_module='detr'`)
Parameters	48.8 M executed at inference (backbone + neck + query head). The checkpoint additionally stores ~16 M parameters for the training-only RPN / RoI / ATSS heads (65.1 M total), which inference never touches.
Classes	80 (COCO `things`)
Max detections / image	100
Training schedule	COCO `train2017`, 12 epochs (1×), AdamW lr 2e-4 (backbone 0.1×), step at epoch 11, AutoAugment multi-scale (short side 480–800, long side ≤1333)
Test-time input	single scale, short side 800 / long side ≤1333, no flip / no TTA

—Paper: DETRs with Collaborative Hybrid Assignments Training, ICCV 2023
—Architecture family: Deformable-DETR (Zhu et al., ICLR 2021) + Co-DETR collaborative training
—Config: `co_deformable_detr_r50_1x_coco.py` (bundled in this repo; identical to the one in the Co-DETR repo under projects/configs/co_deformable_detr/)

What I checked

Integrity and "does it load and run" checks only. This is not an accuracy reproduction:

—✅ Loads with a full state-dict key match in the codetr env (Python 3.8 / torch 1.11.0+cu113 / mmcv-full 1.5.0 / mmdet 2.25.3): zero missing, zero unexpected keys.
—✅ I ran it end-to-end through `tools/inference.py` on a single image, an image folder, and a short video clip (demo/demo.mp4, 12 frames); it produced valid annotated outputs and a frame-indexed detections.json.
—❌ I did not re-measure COCO AP. The 49.5 box AP below is the authors' number, carried from the paper / official model zoo.

Measured inference footprint

Method. I built the model with mmdet.apis.init_detector, then ran inference_detector() on a single image, device='cuda:0', batch size 1, FP32 (no AMP), cudnn.benchmark=False: one warm-up call, then torch.cuda.reset_peak_memory_stats() and 10 timed calls (wall clock, with torch.cuda.synchronize() around each). Memory is the PyTorch caching allocator's own counters: max_memory_allocated includes the resident model weights plus all activation buffers; max_memory_reserved is the allocator's pool high-water mark. Neither includes the CUDA context (~0.3–0.6 GB more in nvidia-smi process RSS for the cu113 runtime). The config's test pipeline resizes to short side 800 / long side ≤ 1333, keep_ratio=True, pad divisor 1.

Item	Value
GPU	NVIDIA GeForce RTX 4070 SUPER, 12 GB, driver 580.173.02
Stack	Python 3.8, torch 1.11.0+cu113 (cu113 wheel; host CUDA toolkit 12.8 unused), mmcv-full 1.5.0, mmdet 2.25.3
Precision / batch	FP32 / 1
Params executed at inference (backbone + neck + query head)	48.8 M
Params stored in the checkpoint (adds the training-only RPN / RoI / ATSS heads)	65.1 M

The checkpoint stores 65.1 M parameters, but roughly 16 M of those belong to the auxiliary training heads and are never used during inference; the deployed detector is 48.8 M. Neither figure is the file size on disk (249 MiB / 260,984,633 bytes, see Provenance).

Source image	Resized `(H×W)` actually fed to the backbone	Peak `max_memory_allocated`	Peak `max_memory_reserved`	Latency / image (warm, n=10)
`demo/demo.jpg` (427×640)	800 × 1199	581 MiB (0.61 GB)	670 MiB (0.70 GB)	mean 60 ms (59–62)
4K source frame (2160×3840)	750 × 1333 (max test size)	598 MiB (0.63 GB)	740 MiB (0.78 GB)	mean 85 ms (84–89)

Activation memory is nearly flat across resolution (deformable attention is sparse), so peak allocator use stays under ~0.65 GB for any single-image input at the default test scale; even counting the CUDA context the process stays well under 2 GB. Latency scales with pixel count (~60 ms at 800×1199 to ~85 ms at the 750×1333 max test size).

CPU inference (--device cpu) also works and needs no GPU; it is substantially slower (tens of seconds per image, not benchmarked here).

Evaluation (as reported by the original authors)

Benchmark	Metric	Value	Source	Reproduced here?
COCO `val2017`	box AP	49.5	Co-DETR paper (arXiv:2211.12860) / official model zoo	No

Schedule: 12 epochs (1×), 300 object queries, single-scale test, no TTA.

For context, the paper reports roughly +3 box AP over the plain Deformable-DETR two-stage R50 baseline at the same 12-epoch schedule; that gap is the contribution of the collaborative auxiliary heads during training. I have not independently re-measured these numbers (no COCO evaluation run); treat them as the authors' claim.

Training data

—COCO 2017 train2017: 118k images, 80 things categories. Lin et al., Microsoft COCO: Common Objects in Context, ECCV 2014 (arXiv:1405.0312).
—Backbone initialization: ResNet-50 pretrained on ImageNet-1k (torchvision://resnet50).

I do not redistribute COCO here. Get it from cocodataset.org. Unlike the Swin-L o365to* checkpoints in the Co-DETR zoo, this checkpoint was trained using COCO and ImageNet initialization only; no Objects365 pretraining is described for it.

Intended use & limitations

Intended use. Research and engineering work that needs a COCO-pretrained object detector in the Co-DETR / Deformable-DETR family: as a baseline, a feature/detection backbone, or a starting point for fine-tuning on a custom COCO-format dataset.

Limitations.

—Trained and evaluated on COCO only; predicts the 80 COCO `thing` categories and nothing else.
—Accuracy can degrade substantially on domain-shifted imagery (aerial, medical, document, heavy weather, non-natural images, etc.).
—The reported 49.5 box AP is the authors' number; I have not reproduced it (see Evaluation).
—No real-time claim. See measured latency for actual timings I got on one GPU.
—Runs only on the legacy OpenMMLab 1.x stack (MMDetection 2.25.3 / MMCV-full 1.5.0 / PyTorch 1.11); no transformers / pipeline() support, and inference: false on the Hub.
—The auxiliary training heads in the checkpoint are not exposed as inference outputs; only the DETR query head's detections are returned.

License status

The Co-DETR source code is MIT-licensed (© 2022 SenseTime X-Lab, LICENSE). The upstream repository does not provide a separate, explicit license for this checkpoint's weights. The original authors have published other Co-DETR checkpoints (the ViT-L family) on Hugging Face with MIT metadata, but I do not treat that as definitive evidence that these ResNet-50 weights are independently licensed under MIT.

Because the applicable rights are undetermined, I set the Hub metadata for this repo to license: unknown. Verify the applicable rights yourself before redistribution or commercial use. For anything beyond local research/evaluation, the safe path is to obtain the weights from, or confirm terms with, the original authors.

This checkpoint was trained using COCO and ImageNet initialization only; no Objects365 pretraining is described for it (unlike the Swin-L o365to* checkpoints). Any applicable dataset terms are therefore COCO's and ImageNet's. Review them yourself if that matters for your use.

If you are one of the original authors and want this mirror removed or transferred, open an issue on this repo or contact dronefreak and I will action it.

Citation

If you use this model, cite the original work:

bibtex

@inproceedings{zong2023detrs,
  title={DETRs with Collaborative Hybrid Assignments Training},
  author={Zong, Zhuofan and Song, Guanglu and Liu, Yu},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  pages={6748--6758},
  year={2023}
}

The underlying detector and backbone:

bibtex

@inproceedings{zhu2021deformable,
  title={Deformable DETR: Deformable Transformers for End-to-End Object Detection},
  author={Zhu, Xizhou and Su, Weijie and Lu, Lewei and Li, Bin and Wang, Xiaogang and Dai, Jifeng},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2021}
}

@inproceedings{he2016deep,
  title={Deep Residual Learning for Image Recognition},
  author={He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={770--778},
  year={2016}
}

Built with MMDetection / MMCV (OpenMMLab).

Acknowledgements

I sincerely thank Zhuofan Zong, Guanglu Song, and Yu Liu for developing Co-DETR and releasing the pretrained weights, and the OpenMMLab team for MMDetection and MMCV. This mirror only makes one of their existing checkpoints easier and more reliable to obtain.

Provenance

co_deformable_detr_r50_1x_coco.pth, 260,984,633 bytes, SHA-256 908d8487f3968f4f93c575478082ef717bbec780e5757d93441c9b66b597726f. I copied it byte-for-byte and unmodified from the authors' Google Drive folder 1asWoZ3SuM6APTL9D-QUF_YW9mjULNdh9.

Repo files: co_deformable_detr_r50_1x_coco.pth (weights), co_deformable_detr_r50_1x_coco.py (the MMDetection config needed to build the model), config.json (metadata summary; also what the Hub uses to count downloads), plus assets/demo_banner.mp4 / assets/demo_banner_poster.jpg for the card. config.json is descriptive only, inference: false, and is not a transformers config.