Mashood/Language_Tactile

<h1 align="center">Language-Guided Representation Learning<br>for Robust Cross-Sensor Tactile Perception</h1>

<p align="center"> <img src="https://img.shields.io/badge/IROS2026-Accepted-164E63" alt="Accepted at IROS 2026"> <a href="https://github.com/Mashood3624/LanguageTactile/blob/main/environment.yml"><img src="https://img.shields.io/badge/Python-3.10-3776AB?logo=python&amp;logoColor=white" alt="Python 3.10"></a> <a href="https://github.com/Mashood3624/Language_Tactile/blob/main/environment.yml"><img src="https://img.shields.io/badge/PyTorch-2.0.1-EE4C2C?logo=pytorch&amp;logoColor=white" alt="PyTorch 2.0.1"></a> </p>

<p align="center"> <a href="https://www.linkedin.com/in/mashood3624/">Mashood M. Mohsan</a> · Muhayy Ud Din · Binzhao Xu · Ahmad Abubakar · Irfan Hussain<br> Khalifa University Center for Autonomous Robotic Systems (KUCARS)<br> Khalifa University, UAE </p>

<p align="center"> <a href="https://www.ku.ac.ae/">Khalifa University</a> · <a href="https://touchripe.com/">TouchRIPE</a> · <a href="https://www.ku.ac.ae/kucars">KUCARS</a> · <a href="https://www.linkedin.com/company/ihlab/">AERIS Lab</a> </p>

<p align="center"> <a href="https://mashood3624.github.io/Language_Tactile/">Project website</a> · <a href="https://youtu.be/QaMzg2h5LKA">Video</a> · <a href="#quick-start">Quick start</a> · <a href="#dataset">Dataset</a> · <a href="#citation">Citation</a> </p>

Language descriptions guide a tactile image encoder to learn material representations across sensors. Training has two stages: distill a frozen BART language teacher into a ViT tactile student, then freeze the tactile encoder and train a material classifier. Inference uses tactile images alone.

[image]

Dataset

Language-Guided Tactile Perception is a multimodal tactile dataset combining data from TVL/HCT and SSVTP. Each experiment sample contains paired tactile and visual observations together with a natural-language description and a material label.

The dataset is intended for research in tactile perception, multimodal representation learning, cross-sensor generalization, material recognition, and language-guided tactile learning.

Dataset Contents

Modalities: Tactile images · RGB images · Tactile–vision pairs · Natural-language descriptions Annotations: Material labels · 32 material categories Splits: Train / evaluation · Distillation · Few-shot classification

Dataset composition

The four experiment CSVs contain 39,717 tactile–vision pairs used in the experiments.

The experiment splits are further divided as follows:

Data organization

After extracting dataset.zip, the dataset is organized as:

dataset/
├── images/
│   ├── hct/
│   └── ssvtp/
├── splits/
└── additional/

Citation

@misc{mohsan2026languageguidedrepresentationlearningrobust,
      title={Language-Guided Representation Learning for Robust Cross-Sensor Material Recognition}, 
      author={Mashood M. Mohsan and Muhayy Ud Din and Binzhao Xu and Ahmad Abubakar and Irfan Hussain},
      year={2026},
      eprint={2609.14783},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2609.14783}, 
}

Acknowledgements

Data sources: TVL/HCT and SSVTP. Code builds on MDistiller, Transformers and the distillation example.