CoolFace
Datasetpublic

humair025/hashed_data

Munch Hashed Index - Lightweight Audio Reference Dataset πŸ“– Overview Munch Hashed Index is a lightweight reference dataset that provides SHA-256 hashes for all audio files in the Munch Urdu TTS Dataset. Instead of storing 1.27 TB of raw audio, this index stores only metadata and cryptographic hashes, enabling: βœ… Fast duplicate detection across 4.17 million audio samples βœ… Efficient dataset exploration without downloading terabytes βœ… Quick metadata queries… See the full description on the dataset page: https://huggingface.co/datasets/humair025/hashed_data.

sourceHugging Facecc-by-4.0updated 10mo agoView on Hugging Face
0likes1kdownloads
Dataset Card

Munch Hashed Index - Lightweight Audio Reference Dataset

![Original Dataset](https://huggingface.co/datasets/humair025/Munch) ![Hashed Index](https://huggingface.co/datasets/humair025/hashed_data) ![Size]() ![Original Size]() ![Space Saved]()

πŸ“– Overview

Munch Hashed Index is a lightweight reference dataset that provides SHA-256 hashes for all audio files in the Munch Urdu TTS Dataset. Instead of storing 1.27 TB of raw audio, this index stores only metadata and cryptographic hashes, enabling:

  • β€”βœ… Fast duplicate detection across 4.17 million audio samples
  • β€”βœ… Efficient dataset exploration without downloading terabytes
  • β€”βœ… Quick metadata queries (voice distribution, text stats, etc.)
  • β€”βœ… Selective audio retrieval - download only what you need
  • β€”βœ… Storage efficiency - 99.92% space reduction (1.27 TB β†’ ~1 GB)

πŸ”— Related Datasets

🎯 What Problem Does This Solve?

The Challenge

The original Munch dataset contains:

  • β€”πŸ“Š 4,167,500 audio-text pairs
  • β€”πŸ’Ύ 1.27 TB total size
  • β€”πŸ“¦ ~8,300 separate parquet files

This makes it difficult to:

  • β€”βŒ Quickly check if specific audio exists
  • β€”βŒ Find duplicate audio samples
  • β€”βŒ Explore metadata without downloading everything
  • β€”βŒ Work on limited bandwidth/storage

The Solution

This hashed index provides:

  • β€”βœ… All metadata (text, voice, timestamps) without audio bytes
  • β€”βœ… SHA-256 hashes for every audio file (unique fingerprint)
  • β€”βœ… File references (which parquet contains each audio)
  • β€”βœ… Fast queries - search 4.17M records in seconds
  • β€”βœ… Retrieve on demand - download only specific audio when needed

πŸš€ Quick Start

Installation

bash
pip install datasets pandas

Basic Usage

python
from datasets import load_dataset
import pandas as pd

# Load the entire hashed index (fast - only ~1 GB!)
ds = load_dataset("humair025/hashed_data", split="train")
df = pd.DataFrame(ds)

print(f"Total records: {len(df)}")
print(f"Unique audio hashes: {df['audio_bytes_hash'].nunique()}")
print(f"Voices: {df['voice'].unique()}")

Find Duplicates

python
# Check for duplicate audio
duplicates = df[df.duplicated(subset=['audio_bytes_hash'], keep=False)]

if len(duplicates) > 0:
    print(f"⚠️  Found {len(duplicates)} duplicate rows")
    print(f"   Unique audio files: {df['audio_bytes_hash'].nunique()}")
    print(f"   Redundancy: {(1 - df['audio_bytes_hash'].nunique()/len(df))*100:.2f}%")
else:
    print("βœ… No duplicates found!")

Search by Voice

python
# Find all "ash" voice samples
ash_samples = df[df['voice'] == 'ash']
print(f"Ash voice samples: {len(ash_samples)}")

# Get file containing first ash sample
first_ash = ash_samples.iloc[0]
print(f"File: {first_ash['parquet_file_name']}")
print(f"Text: {first_ash['text']}")

Search by Text

python
# Find audio for specific text
query = "یہ ایک Ω†Ω…ΩˆΩ†Ϋ"
matches = df[df['text'].str.contains(query, na=False)]
print(f"Found {len(matches)} matches")

Retrieve Original Audio

python
from datasets import load_dataset as load_original
import numpy as np
from scipy.io import wavfile
import io

def get_audio_by_hash(audio_hash, index_df):
    """Retrieve original audio bytes using the hash"""
    # Find the row with this hash
    row = index_df[index_df['audio_bytes_hash'] == audio_hash].iloc[0]
    
    # Download only the specific parquet file containing this audio
    ds = load_original(
        "humair025/Munch",
        data_files=[row['parquet_file_name']],
        split="train"
    )
    
    # Find matching row by ID
    for audio_row in ds:
        if audio_row['id'] == row['id']:
            return audio_row['audio_bytes']
    
    return None

# Example: Get audio for first row
row = df.iloc[0]
audio_bytes = get_audio_by_hash(row['audio_bytes_hash'], df)

# Convert to WAV and play
def pcm16_to_wav(pcm_bytes, sample_rate=22050):
    audio_array = np.frombuffer(pcm_bytes, dtype=np.int16)
    wav_io = io.BytesIO()
    wavfile.write(wav_io, sample_rate, audio_array)
    wav_io.seek(0)
    return wav_io

wav_io = pcm16_to_wav(audio_bytes)
# In Jupyter: IPython.display.Audio(wav_io, rate=22050)

πŸ“Š Dataset Structure

Data Fields

FieldTypeDescription
idintOriginal paragraph ID from source dataset
parquet_file_namestringSource file in Munch dataset
textstringOriginal Urdu text
transcriptstringTTS transcript (may differ from input)
voicestringVoice used (alloy, echo, fable, onyx, nova, shimmer, coral, verse, ballad, ash, sage, amuch, dan)
audio_bytes_hashstringSHA-256 hash of audio_bytes (64 hex chars)
audio_size_bytesintSize of original audio in bytes
timestampstringISO timestamp of generation (nullable)
errorstringError message if generation failed (nullable)

Example Row

python
{
    'id': 42,
    'parquet_file_name': 'tts_data_20251203_130314_83ab0706.parquet',
    'text': 'یہ ایک Ω†Ω…ΩˆΩ†Ϋ Ω…ΨͺΩ† ہے۔',
    'transcript': 'یہ ایک Ω†Ω…ΩˆΩ†Ϋ Ω…ΨͺΩ† ہے۔',
    'voice': 'ash',
    'audio_bytes_hash': 'a3f7b2c8e9d1f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9',
    'audio_size_bytes': 52340,
    'timestamp': '2025-12-03T13:03:14.123456',
    'error': None
}

🎯 Use Cases

1. Dataset Quality Analysis

python
# Check for duplicates
unique_ratio = df['audio_bytes_hash'].nunique() / len(df)
print(f"Unique audio ratio: {unique_ratio*100:.2f}%")

# Analyze voice distribution
voice_dist = df['voice'].value_counts()
print(voice_dist)

# Find failed generations
failed = df[df['error'].notna()]
print(f"Failed generations: {len(failed)}")

2. Efficient Data Exploration

python
# Browse dataset without downloading audio
print(df[['id', 'text', 'voice', 'audio_size_bytes']].head(20))

# Filter by criteria
short_audio = df[df['audio_size_bytes'] < 30000]
long_text = df[df['text'].str.len() > 200]

3. Selective Download

python
# Download only specific voices
ash_files = df[df['voice'] == 'ash']['parquet_file_name'].unique()
ds = load_dataset("humair025/Munch", data_files=list(ash_files))

# Download only short audio samples
small_files = df[df['audio_size_bytes'] < 40000]['parquet_file_name'].unique()
ds = load_dataset("humair025/Munch", data_files=list(small_files[:10]))

4. Deduplication Pipeline

python
# Create deduplicated subset
df_unique = df.drop_duplicates(subset=['audio_bytes_hash'], keep='first')
print(f"Original: {len(df)} rows")
print(f"Unique: {len(df_unique)} rows")
print(f"Duplicates removed: {len(df) - len(df_unique)}")

# Save unique references
df_unique.to_parquet('unique_audio_index.parquet')

5. Audio Similarity Search

python
# Find audio with similar hash prefixes (for clustering)
target_hash = df.iloc[0]['audio_bytes_hash']
prefix = target_hash[:8]

similar = df[df['audio_bytes_hash'].str.startswith(prefix)]
print(f"Similar audio candidates: {len(similar)}")

πŸ“ˆ Dataset Statistics

Size Comparison

MetricOriginal DatasetHashed IndexReduction
Total Size1.27 TB~1 GB99.92%
Records4,167,5004,167,500Same
Files~8,300 parquetConsolidated~8,300Γ— fewer
Download Time (100 Mbps)~28 hours~90 seconds~1,100Γ—
Load TimeMinutes-HoursSeconds~100Γ—
Memory UsageCannot fit in RAM~2-3 GB RAMFits easily

Content Statistics

πŸ“Š Dataset Overview:
   Total Records: 4,167,500
   Total Files: ~8,300 parquet files
   Voices: 13 (alloy, echo, fable, onyx, nova, shimmer, coral, verse, ballad, ash, sage, amuch, dan)
   Language: Urdu (primary)
   Avg Audio Size: ~50-60 KB per sample
   Avg Duration: ~3-5 seconds per sample
   Total Duration: ~3,500-5,800 hours of audio

πŸ”§ Advanced Usage

Batch Analysis

python
# Analyze all hash files
from datasets import load_dataset

ds = load_dataset("humair025/hashed_data", split="train")
df = pd.DataFrame(ds)

# Group by voice
voice_stats = df.groupby('voice').agg({
    'id': 'count',
    'audio_size_bytes': 'mean',
    'audio_bytes_hash': 'nunique'
}).rename(columns={
    'id': 'total_samples',
    'audio_size_bytes': 'avg_size',
    'audio_bytes_hash': 'unique_audio'
})

print(voice_stats)

Cross-Reference with Original

python
# Check if a hash exists in original dataset
def verify_hash_exists(audio_hash, parquet_file):
    """Verify a hash actually exists in the original dataset"""
    from datasets import load_dataset
    import hashlib
    
    ds = load_dataset(
        "humair025/Munch",
        data_files=[parquet_file],
        split="train"
    )
    
    for row in ds:
        computed_hash = hashlib.sha256(row['audio_bytes']).hexdigest()
        if computed_hash == audio_hash:
            return True
    
    return False

# Verify first entry
first_row = df.iloc[0]
exists = verify_hash_exists(
    first_row['audio_bytes_hash'],
    first_row['parquet_file_name']
)
print(f"Hash verified: {exists}")

Export Unique Dataset

python
# Create a new dataset with only unique audio
df_unique = df.drop_duplicates(subset=['audio_bytes_hash'], keep='first')

# Get list of unique parquet files needed
unique_files = df_unique['parquet_file_name'].unique()

print(f"Unique audio samples: {len(df_unique)}")
print(f"Files needed: {len(unique_files)} out of {df['parquet_file_name'].nunique()}")

# Calculate space savings
original_size = len(df) * df['audio_size_bytes'].mean()
unique_size = len(df_unique) * df_unique['audio_size_bytes'].mean()
savings = (1 - unique_size/original_size) * 100

print(f"Space savings: {savings:.2f}%")

πŸ› οΈ How This Index Was Created

This dataset was generated using an automated pipeline:

Processing Pipeline

  1. 1.Batch Download: Download 40 parquet files at a time from source
  2. 2.Hash Computation: Compute SHA-256 for each audio_bytes field
  3. 3.Metadata Extraction: Extract text, voice, and other metadata
  4. 4.Save & Upload: Save hash file, upload to HuggingFace
  5. 5.Clean Up: Delete local cache to save disk space
  6. 6.Resume: Track processed files, skip already-processed

Pipeline Features

  • β€”βœ… Resumable: Checkpoint system tracks progress
  • β€”βœ… Memory Efficient: Processes in batches, clears cache
  • β€”βœ… Error Tolerant: Skips corrupted files, continues processing
  • β€”βœ… No Duplicates: Checks target repo to avoid reprocessing
  • β€”βœ… Automatic Upload: Streams results to HuggingFace

Technical Details

python
# Hash computation
import hashlib
hash = hashlib.sha256(audio_bytes).hexdigest()

# Batch size: 40 files per batch
# Processing time: ~4-6 hours for full dataset
# Output: Multiple hashed_*.parquet files

πŸ“Š Performance Metrics

Query Performance

python
import time

# Load index
start = time.time()
ds = load_dataset("humair025/hashed_data", split="train")
df = pd.DataFrame(ds)
print(f"Load time: {time.time() - start:.2f}s")

# Query by hash
start = time.time()
result = df[df['audio_bytes_hash'] == 'target_hash']
print(f"Hash lookup: {(time.time() - start)*1000:.2f}ms")

# Query by voice
start = time.time()
result = df[df['voice'] == 'ash']
print(f"Voice filter: {(time.time() - start)*1000:.2f}ms")

Expected Performance:

  • β€”Load full dataset: 10-30 seconds
  • β€”Hash lookup: < 10 milliseconds
  • β€”Voice filter: < 50 milliseconds
  • β€”Full dataset scan: < 5 seconds

πŸ”— Integration with Original Dataset

Workflow Example

python
# 1. Query the index (fast)
df = pd.DataFrame(load_dataset("humair025/hashed_data", split="train"))
target_rows = df[df['voice'] == 'ash'].head(100)

# 2. Get unique parquet files
files_needed = target_rows['parquet_file_name'].unique()

# 3. Download only needed files (selective)
from datasets import load_dataset
ds = load_dataset(
    "humair025/Munch",
    data_files=list(files_needed),
    split="train"
)

# 4. Match by ID to get audio
for idx, row in target_rows.iterrows():
    for audio_row in ds:
        if audio_row['id'] == row['id']:
            # Process audio_bytes
            audio = audio_row['audio_bytes']
            break

πŸ“œ Citation

If you use this dataset in your research, please cite both the original dataset and this index:

BibTeX

bibtex
@dataset{munch_hashed_index_2025,
  title={Munch Hashed Index: Lightweight Reference Dataset for Urdu TTS},
  author={Munir, Humair},
  year={2025},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/datasets/humair025/hashed_data}},
  note={Index of humair025/Munch dataset with SHA-256 audio hashes}
}

@dataset{munch_urdu_tts_2025,
  title={Munch: Large-Scale Urdu Text-to-Speech Dataset},
  author={Munir, Humair},
  year={2025},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/datasets/humair025/Munch}}
}

APA Format

Munir, H. (2025). Munch Hashed Index: Lightweight Reference Dataset for Urdu TTS 
[Dataset]. Hugging Face. https://huggingface.co/datasets/humair025/hashed_data

Munir, H. (2025). Munch: Large-Scale Urdu Text-to-Speech Dataset [Dataset]. 
Hugging Face. https://huggingface.co/datasets/humair025/Munch

MLA Format

Munir, Humair. "Munch Hashed Index: Lightweight Reference Dataset for Urdu TTS." 
Hugging Face, 2025, https://huggingface.co/datasets/humair025/hashed_data.

Munir, Humair. "Munch: Large-Scale Urdu Text-to-Speech Dataset." Hugging Face, 2025, 
https://huggingface.co/datasets/humair025/Munch.

🀝 Contributing

Report Issues

Found a problem? Please open an issue:

  • β€”Missing hash files
  • β€”Incorrect metadata
  • β€”Hash mismatches
  • β€”Documentation improvements

Suggest Improvements

We welcome suggestions for:

  • β€”Additional metadata fields
  • β€”Better indexing strategies
  • β€”Integration examples
  • β€”Use case documentation

πŸ“„ License

This index dataset inherits the license from the original Munch dataset:

Creative Commons Attribution 4.0 International (CC-BY-4.0)

You are free to:

  • β€”βœ… Share β€” copy and redistribute
  • β€”βœ… Adapt β€” remix, transform, build upon
  • β€”βœ… Commercial use β€” use commercially

Under the terms:

  • β€”πŸ“ Attribution β€” Give appropriate credit to original dataset

πŸ”— Important Links

❓ FAQ

Q: Why use hashes instead of audio?

A: Hashes provide unique fingerprints for audio files while taking only 64 bytes vs ~50KB per audio. This enables duplicate detection and fast queries without storing massive audio files.

Q: Can I reconstruct audio from hashes?

A: No. SHA-256 is a one-way cryptographic hash. You must download the original audio from the Munch dataset using the file reference provided.

Q: How accurate are the hashes?

A: SHA-256 has virtually zero collision probability. If two hashes match, the audio is identical (byte-for-byte).

Q: How do I get the actual audio?

A: Use the parquet_file_name and id fields to locate and download the specific audio from the original dataset. See examples above.

Q: Is this dataset complete?

A: Yes, this index covers all 4,167,500 rows across all ~8,300 parquet files from the original Munch dataset.

Q: What's the difference between this and Munch-1 Index?

A: This indexes the original Munch dataset (1.27 TB, 4.17M samples). The Munch-1 Index indexes the newer Munch-1 dataset (3.28 TB, 3.86M samples).

Q: Can I contribute?

A: Yes! Help verify hashes, report inconsistencies, or suggest improvements via discussions.

πŸ™ Acknowledgments

  • β€”Original Dataset: humair025/Munch
  • β€”TTS Generation: OpenAI-compatible models
  • β€”Voices: 13 high-quality voices (alloy, echo, fable, onyx, nova, shimmer, coral, verse, ballad, ash, sage, amuch, dan)
  • β€”Infrastructure: HuggingFace Datasets platform
  • β€”Hashing: SHA-256 cryptographic hash function

πŸ“ Version History

  • β€”v1.0.0 (December 2025): Initial release
  • β€”Processed all ~8,300 parquet files
  • β€”4,167,500 audio samples indexed
  • β€”SHA-256 hashes computed for all audio
  • β€”~99.92% space reduction achieved

Last Updated: December 2025

Status: βœ… Complete

πŸ’‘ Pro Tip: Start with this lightweight index to explore the dataset, then selectively download only the audio you need from the original Munch dataset!