Datasetpublic

mesolitica/unsupervised-malay-youtube-speaker-diarization

Unsupervised malay speakers from youtube videos 10492 unique speakers with at least 75 hours of voice activities. Steps to reproduce at https://github.com/huseinzol05/malaya-speech/blob/master/data/youtube/process-youtube.ipynb how-to Download and extract processed-youtube.tar.gz, each processed videos saved as pickle, {video_name}.pkl. Each pickle file got, [{'wav_data':… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/unsupervised-malay-youtube-speaker-diarization.

sourceHugging Faceupdated 4y agoView on Hugging Face

0likes10downloads

Dataset Card

Unsupervised malay speakers from youtube videos

10492 unique speakers with at least 75 hours of voice activities. Steps to reproduce at https://github.com/huseinzol05/malaya-speech/blob/master/data/youtube/process-youtube.ipynb

how-to

1.Download and extract processed-youtube.tar.gz, each processed videos saved as pickle, {video_name}.pkl.

1.Each pickle file got,

python

[{'wav_data': '/home/husein/ssd2/processed-youtube-v2/"Abam_peluk_saya_lama_atas_pentas_akhir_MLM"-_Ali_Puteh_menangis_imbau_saat_manis_dengan_arwah_abang-_MdgGr7VD7w/0.mp3',
  'timestamp': datetime.datetime(2023, 3, 2, 18, 45, 45, 778042),
  'asr_model': ('kenapa tak mahu bangun kau abang',
   [0.5325799628135358],
   [309, 9, 399, 633, 108, 252]),
  'classification_model': (array([ 3.02432757e-03, -3.64390127e-02,  2.93319039e-02, -2.84599233e-02,
          -5.04244901e-02,  6.03185333e-02,  7.04260264e-03,  7.36895157e-03,
           2.41034012e-02, -3.31214964e-02, -1.61228217e-02, -1.92081463e-02,
          -1.77928973e-02,  1.05488757e-02,  5.11314301e-03,  2.08497643e-02,
           2.80407351e-02, -1.34683009e-02,  1.10213496e-02, -5.76948654e-03,
           2.11171638e-02, -3.10498872e-03,  1.60899870e-02, -2.22061612e-02,
          -3.09270490e-02,  1.03673469e-02,  2.29822248e-02,  5.44358939e-02,
          -9.44061391e-03,  3.24469656e-02, -1.40673192e-02,  6.55731931e-03,
           1.94134321e-02,  2.31755860e-02, -8.62774719e-03, -3.72681394e-03,
          -3.17485556e-02, -1.12474747e-02,  1.65595114e-02,  2.31244415e-02,
           3.28784771e-02,  8.52510054e-03, -6.41896739e-04,  3.13562714e-03,
          -3.15982029e-02,  1.72785181e-03,  1.58039071e-02, -9.93900001e-03,
           2.03248486e-02, -2.98949536e-02,  3.53759155e-02,  3.06809470e-02,
          -3.68881435e-03, -3.98267582e-02, -2.07101982e-02,  2.51877047e-02,
          -2.51530181e-03,  1.06034977e-02,  1.24978041e-02,  2.35916697e-03,
           1.31300613e-02, -1.62451845e-02, -2.09861826e-02,  3.17490734e-02,
          -1.18532358e-02,  4.25735563e-02,  4.17908467e-02,  1.21251179e-03,
          -3.85571155e-03, -9.50544327e-03, -7.37808086e-03,  2.63940021e-02,
           1.09219365e-02,  3.05683501e-02, -4.08848785e-02, -1.71920974e-02,
          -1.46033484e-02, -3.29053291e-05,  3.84788848e-02, -7.86552951e-03,
           1.01251132e-03,  2.72140447e-02,  2.52339337e-02,  3.39004360e-02,
          -1.38184745e-02,  2.60320995e-02, -1.01425601e-02, -1.16012329e-02,
           4.30319924e-03, -1.01203052e-02, -4.66396799e-03, -2.64480542e-02,
           3.44322808e-02, -4.64622118e-03,  1.06053520e-02,  1.37923108e-02,
          -2.05409434e-03, -1.19995829e-02,  2.10450366e-02, -2.87155900e-03,
          -1.39515549e-02, -1.51185887e-02,  2.29053162e-02, -1.78178120e-02,
           1.95855577e-03,  2.37271357e-02,  2.80657201e-03, -6.08753460e-03,
          -2.01220363e-02,  3.22612897e-02,  1.82474777e-02,  5.31493872e-02,
          -7.08705634e-02,  2.76431069e-03,  1.03597697e-02, -3.53837833e-02,
           1.38167264e-02, -5.91275143e-03,  1.84398554e-02,  6.05177172e-02,
           1.14565976e-02,  1.56977493e-02, -1.82731878e-02, -4.58574407e-02,
          -1.08330613e-02, -1.16500622e-02, -1.19803764e-04,  6.48374185e-02,
          -1.21538760e-03, -5.41793741e-02,  1.38867721e-02,  3.52845751e-02,
          -2.08288375e-02,  1.03750750e-02, -2.17110049e-02,  2.29265504e-02,
          -1.21381739e-02, -1.47071329e-03, -4.36875001e-02, -2.25690063e-02,
          -4.16939743e-02, -8.39853752e-03, -2.06098761e-02,  2.30504461e-02,
           3.48615423e-02, -4.18495797e-02, -2.41985917e-03, -3.18994140e-03,
           1.22078639e-02, -9.50168632e-03, -1.97298196e-03,  1.30731370e-02,
           2.07234323e-02,  1.08521534e-02,  2.30542179e-02, -2.54045837e-02,
           1.45645533e-02, -1.08493539e-02, -1.30415503e-02,  3.29123251e-02,
           3.46204527e-02,  2.58748885e-04, -1.28235819e-03, -1.32823242e-02,
           5.47284493e-03, -2.62062326e-02,  2.31803600e-02, -2.04505119e-02,
           2.32407395e-02,  2.12946888e-02, -1.28869051e-02, -6.81399694e-03,
           5.68802692e-02,  4.31004271e-04,  1.67261921e-02,  2.93559525e-02,
           1.32581135e-02, -9.03073605e-03, -9.38207190e-03,  1.74718127e-02,
           1.72506981e-02,  5.02267219e-02, -1.32851647e-02,  5.07321544e-02,
          -1.87530685e-02,  4.18599546e-02,  1.50075918e-02, -2.61102356e-02,
          -1.59594957e-02,  1.36823149e-03, -9.64679196e-03,  1.71130225e-02],
         dtype=float32),
   'speaker 0')}]

—all mp3 files postprocessing using https://malaya-speech.readthedocs.io/en/latest/load-noise-reduction.html and https://malaya-speech.readthedocs.io/en/latest/load-speech-enhancement.html
—wav_data is directory of the audio, prune the path to proper extracted directory.
—asr_model is predicted using the best model that we have, conformer-medium, returned (text, probability, subwords), https://malaya-speech.readthedocs.io/en/latest/load-stt-transducer-model-pt.html
—classification_model is predicted using NEMO TITANET Large speaker verification model, https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/titanet_large, with streaming speaker similarity, https://malaya-speech.readthedocs.io/en/latest/huggingface-repository.html

1.Group by similar speakers using pagerank method (scipy.sparse.linalg.gmres),

—90% similar, from 10492 unique speakers become 6085 unique speakers, https://github.com/huseinzol05/malaya-speech/blob/master/data/youtube/mapping-youtube-speakers-90.json
—85% similar, from 10492 unique speakers become 4312 unique speakers, https://github.com/huseinzol05/malaya-speech/blob/master/data/youtube/mapping-youtube-speakers-85.json
—80% similar, from 10492 unique speakers become 2912 unique speakers, https://github.com/huseinzol05/malaya-speech/blob/master/data/youtube/mapping-youtube-speakers-80.json

Speaker name defined as,

python

import os
import pickle

pkl = 'filename.pkl'
with open(pkl, 'rb') as fopen:
  data = pickle.load(fopen)

filename = os.path.split(pkl)[1].replace('.pkl', '')
for result in data:
  speaker_name = f'{filename}-{speaker}'
  actual_speaker = unique_speakers[speaker_name]

Check example at https://github.com/huseinzol05/malaya-speech/blob/master/data/youtube/calculate-lengths-80.ipynb