Mashood/Language_Tactile

1---2task_categories:3- image-classification4- image-text-to-image5- image-to-text6- text-classification7- text-to-image8language:9- en10tags:11- tactile12- tactile-sensing13- touch14- vision15- language16- multimodal17- touch-vision-language18- robotics19- material-recognition20- material-classification21- cross-sensor22- cross-modal23- representation-learning24- knowledge-distillation25- vision-transformer26- digit-sensor27- hct28- ssvtp29- VBTS30pretty_name: Language-Guided Tactile Perception Dataset31size_categories:32- 10K<n<100K33---34<h1 align="center">Language-Guided Representation Learning<br>for Robust Cross-Sensor Tactile Perception</h1>35 36<p align="center">37  <img src="https://img.shields.io/badge/IROS_2026-Accepted-164E63" alt="Accepted at IROS 2026">38  <a href="https://github.com/Mashood3624/Language_Tactile/blob/main/environment.yml"><img src="https://img.shields.io/badge/Python-3.10-3776AB?logo=python&amp;logoColor=white" alt="Python 3.10"></a>39  <a href="https://github.com/Mashood3624/Language_Tactile/blob/main/environment.yml"><img src="https://img.shields.io/badge/PyTorch-2.0.1-EE4C2C?logo=pytorch&amp;logoColor=white" alt="PyTorch 2.0.1"></a>40</p>41 42<p align="center">43  <a href="https://www.linkedin.com/in/mashood3624/">Mashood M. Mohsan</a> · Muhayy Ud Din · Binzhao Xu · Ahmad Abubakar · Irfan Hussain<br>44  Khalifa University Center for Autonomous Robotic Systems (KUCARS)<br>45  Khalifa University, UAE46</p>47 48<p align="center">49  <a href="https://www.ku.ac.ae/">Khalifa University</a> ·50  <a href="https://touchripe.com/">TouchRIPE</a> ·51  <a href="https://www.ku.ac.ae/kucars">KUCARS</a> ·52  <a href="https://www.linkedin.com/company/ihlab/">AERIS Lab</a>53</p>54 55<p align="center">56  <a href="https://mashood3624.github.io/Language_Tactile/">Project website</a> ·57  <a href="https://youtu.be/QaMzg2h5LKA">Video</a> ·58  <a href="#quick-start">Quick start</a> ·59  <a href="#dataset">Dataset</a> ·60  <a href="#citation">Citation</a>61</p>62 63Language descriptions guide a tactile image encoder to learn material representations across sensors. Training has two stages: distill a frozen BART language teacher into a ViT tactile student, then freeze the tactile encoder and train a material classifier. Inference uses tactile images alone.64 65![Language–tactile distillation and downstream classification](assets/architecture.png)66 67## Dataset68 69**Language-Guided Tactile Perception** is a multimodal tactile dataset combining data from **TVL/HCT** and **SSVTP**. Each experiment sample contains paired tactile and visual observations together with a natural-language description and a material label.70 71The dataset is intended for research in tactile perception, multimodal representation learning, cross-sensor generalization, material recognition, and language-guided tactile learning.72 73### Dataset Contents74 75**Modalities:** Tactile images · RGB images · Tactile–vision pairs · Natural-language descriptions  76**Annotations:** Material labels · 32 material categories  77**Splits:** Train / evaluation · Distillation · Few-shot classification78 79### Dataset composition80 81The four experiment CSVs contain **39,717 tactile–vision pairs** used in the experiments.82 83| Source | Distillation | Few-shot | Total |84|---|---:|---:|---:|85| TVL / HCT | 27,223 | 8,683 | 35,906 |86| SSVTP | 2,475 | 1,336 | 3,811 |87| **Combined** | **29,698** | **10,019** | **39,717** |88 89The experiment splits are further divided as follows:90 91| Task | Train | Evaluation | Total |92|---|---:|---:|---:|93| Language-guided distillation | 21,035 | 8,663 | 29,698 |94| Material classification | 7,147 | 2,872 | 10,019 |95| **Combined** | **28,182** | **11,535** | **39,717** |96 97### Data organization98 99After extracting `dataset.zip`, the dataset is organized as:100 101```text102dataset/103├── images/104│   ├── hct/105│   └── ssvtp/106├── splits/107└── additional/108```109## Citation110 111```bibtex112@misc{mohsan2026languageguidedrepresentationlearningrobust,113      title={Language-Guided Representation Learning for Robust Cross-Sensor Material Recognition}, 114      author={Mashood M. Mohsan and Muhayy Ud Din and Binzhao Xu and Ahmad Abubakar and Irfan Hussain},115      year={2026},116      eprint={2609.14783},117      archivePrefix={arXiv},118      primaryClass={cs.RO},119      url={https://arxiv.org/abs/2609.14783}, 120}121```122 123## Acknowledgements124 125Data sources: [TVL/HCT](https://tactile-vlm.github.io/) and [SSVTP](https://sites.google.com/berkeley.edu/ssvtp). Code builds on [MDistiller](https://github.com/megvii-research/mdistiller), [Transformers](https://github.com/huggingface/transformers) and the [distillation example](https://github.com/philschmid/knowledge-distillation-transformers-pytorch-sagemaker).