Modelpublic

cvlab/new-york-smells-coip

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face

0likes

Model Card

COIP — Contrastive Olfaction-Image Pretraining

Joint embeddings of smell and sight, trained on New York Smells: 7,000 paired image / electronic-nose recordings captured in the wild across New York City with a Cyranose 320 (32 conducting-polymer sensors).

Two towers project into a shared 128-d L2-normalised space, so a dot product between an olfaction embedding and an image embedding is a cosine similarity. Given a smell, you can retrieve the image of what produced it; given the embedding alone, a linear probe can name the scene, object, or material.

—Paper: arXiv:2511.20544
—Code: https://github.com/cvondrick/new-york-smells
—Project page: https://smell.cs.columbia.edu

Usage

python

from nys.models import from_pretrained

model = from_pretrained("transformer")

image_emb, smell_emb = model(image, raw_smell)    # both (B, 128), L2-normalised
similarity = smell_emb @ image_emb.t()

raw_smell is (B, 28, 32) — concat(baseline_raw[:14], sample_raw[:14]), z-scored per sensor with training-split statistics. The smellprint model instead takes the (B, 32) hand-crafted feature. nys.data.SmellDataset assembles either from the released dataset.

Models

Encoder	Input	File	Architecture
`transformer`	raw `(28, 32)`	`transformer/model.pth`	CLS + 6-layer pre-norm transformer
`cnn`	raw `(28, 32)`	`cnn/model.pth`	4 conv blocks over the sensor matrix
`mlp`	raw `(28, 32)`	`mlp/model.pth`	3 × Linear/ReLU/LayerNorm
`smellprint`	`(32,)`	`smellprint/model.pth`	MLP over the smellprint feature

Each is ~96 MB, trained on the object-level split with the image tower finetuned from ImageNet initialisation, and carries its training configuration under legacy_opt.

Cross-modal retrieval

Smell query against a gallery of image embeddings, N = 933.

Smell encoder	Mean rank ↓	Median rank ↓	R@5 ↑	R@10 ↑	R@20 ↑
Chance	467	467	0.54	1.07	2.14
MLP (smellprint)	362.6	324	1.61	2.79	5.68
MLP (raw)	134.4	48	13.50	22.19	33.33
CNN (raw)	125.0	37	17.90	26.69	38.05
Transformer (raw)	106.1	30	17.36	28.72	41.80

Training on the raw olfactory signal substantially outperforms the hand-crafted smellprint on every metric, and the transformer is the strongest of the raw encoders.

Recognising scenes, objects and materials from smell

A linear probe on the frozen olfaction embedding, on the object-level test split. Random-weight encoders are shown for reference.

Encoder	Init	Scene (8)	Material (51)	Object (49)
—	chance	12.5	2.0	2.0
MLP (smellprint)	random	29.5	2.8	3.7
MLP (smellprint)	COIP	32.7	5.0	4.8
MLP (raw)	random	90.6	10.8	17.1
MLP (raw)	COIP	94.2	10.4	17.7
CNN (raw)	random	71.2	9.1	12.6
CNN (raw)	COIP	93.2	10.1	18.4
Transformer (raw)	random	72.2	8.5	10.9
Transformer (raw)	COIP	87.4	14.5	16.1

Scene is top-1; material and object are balanced accuracy (mean per-class recall). Those two tasks have 51 and 49 heavily skewed classes — the largest material class alone is 13.7% of the test set — so unbalanced top-1 there largely measures the class prior. Scene's 8 classes are roughly even, where the two metrics coincide.

Scene labels are per-session; object and material labels are GPT-4o annotations derived from the paired image. All three tasks are well above chance from smell alone.

Scope

60 recording sessions in New York City over roughly three months. Electronic-nose response varies with temperature, humidity and sensor age, so behaviour on other hardware, other climates, or odorants unlike those in the dataset is untested.

Citation

bibtex

@article{ozguroglu2025smell,
  title={New York Smells: A Large Multimodal Dataset for Olfaction},
  author={Ozguroglu, Ege and Liang, Junbang and Liu, Ruoshi and Chiquier, Mia and DeTienne, Michael and Qian, Wesley Wei and Horowitz, Alexandra and Owens, Andrew and Vondrick, Carl},
  journal={arXiv preprint arXiv:2511.20544},
  year={2025}
}

License

CC BY 4.0, matching the dataset.