validation-data
esb-datasets-earnings22-validation-tiny-filteredA filtered (<=30s duration) slice (512 samples) of the Earnings22 dataset.
def add_duration(sample):
y, sr = sample['audio']["array"], sample['audio']["sampling_rate"]
sample['duration_ms']=librosa.get_duration(y=y, sr=sr) * 1000
return sample
tedlium = load_dataset("esb/datasets", "earnings22", split='validation', trust_remote_code=True)
# compute duration to filter
tedlium = tedlium.map(add_duration)
tedlium = tedlium.select(range(512))
# Whisper max supported duration
tedlium… See the full description on the dataset page: https://huggingface.co/datasets/D4nt3/esb-datasets-earnings22-validation-tiny-filtered.Surya-1.0_validation_data
Validation data for Surya 1.0
This dataset comprises imagery from NASA's Solar Dynamics Observatory (SDO). The data can and should be used to validate a local installation of the Surya Foundation Model for Heliophysics. The data is compressed; you should use the hdf5plugin to read it directly.
validation-datasets-for-llama.cppThis repository contains validation datasets for use with the perplexity tool from the llama.cpp project.
Note: PR #5047 is required to be able to use these datasets.
The simple program in demo.cpp shows how to read these files and can be used to combine two files into one.
The simple program in convert.cpp shows how to convert the data to JSON. For instance:
g++ -o convert convert.cpp
./convert arc-easy-validation.bin arc-easy-validation.json
ValidationDataSetttm-validation-datasetlego_assemblies_openpi_validation