RASMUS/Finnish-ASR-Canary-v2
01.2k
1**Speech Data Simulator**2===============3 4Outline5------------6 7The speech data simulator generates synthetic multispeaker audio sessions for training or evaluating models for multispeaker ASR or speaker diarization. This tool aims to address the lack of labelled multispeaker training data and to help models deal with overlapping speech.8 9The simulator loads audio files from different speakers as well as forced alignments for each sentence and concatenates the audio files together to build a synthetic multispeaker audio session. The simulator uses the word alignments to segment the audio from each speaker to produce utterances of the desired length. The simulator also incorporates synthetic room impulse response (RIR) generation in order to simulate multi-microphone multispeaker sessions.10 11Features12------------13 14The simulator is reconfigurable and has several options including:15 16* Amount of overlapping speech 17 - The percentage of overlapping speech out of the total speaker time.18* Percentage of silence19 - The percentage of the overall audio session that has no speakers talking.20* Sentence length distribution21 - The distribution of sentence lengths that is used for sampling (the parameters passed in are for a negative binomial distribution).22* Number of speakers per session23* Length of each session24* Variance in speaker dominance 25 - Determines what portion of the speaking time will be used by each speaker in a session. Increasing this value will make it more likely that a few speakers dominate the conversation. 26* Turn taking 27 - Determines how likely it is that a speaker keeps talking after completing an utterance.28* Background noise 29 30The simulator can be used in two modes: near field (no Room Impulse Response) as well as far field (including synthetic RIR). When using synthetic RIR generation, multiple microphones can be placed in the simulated room environment for multichannel simulations.31 32The simulator also has a speaker enforcement mode which ensures that the correct number of speakers appear in each session (otherwise not guaranteed since speaker turns are stochastic). In speaker enforcement mode, the length of the session or speaker probabilities may be adjusted to ensure all speakers are present.33 34Required Datasets35------------36 37* LibriSpeech (or another single-speaker dataset)38* LibriSpeech word alignments from [here](https://github.com/CorentinJ/librispeech-alignments) (or alignments corresponding to another single-speaker dataset)39 40Example alignment format from the LibriSpeech dataset (to be passed as input to the `scripts/speaker_tasks/create_librispeech_alignment_manifest.py` script):41 42* Alignment files are stored at <Speaker ID>/<Chapter ID>/<Speaker ID>-<Chapter ID>.txt, and each line in the alignment file corresponds to a separate sentence43* Example of a line in `dev-clean/1272/128104/1272-128104.txt': '1272-128104-0000 ",MISTER,QUILTER,IS,THE,APOSTLE,OF,THE,MIDDLE,CLASSES,,AND,WE,ARE,GLAD,TO,WELCOME,HIS,GOSPEL," "0.500,0.800,1.270,1.400,1.520,2.150,2.270,2.350,2.620,3.270,3.300,3.450,3.600,3.670,4.070,4.200,4.600,4.840,5.510,5.855"`44 45Optional Datasets46------------47 48* Room Impulse Response and Noise Database from [here](https://www.openslr.org/resources/28/rirs_noises.zip) (or another background noise dataset)49 50Installation (after installing NeMo)51------------52 53Note that only one of gpuRIR or pyroomacoustics is required for RIR simulation.54 55```bash56pip install cmake57pip install https://github.com/DavidDiazGuerra/gpuRIR/zipball/master58pip install pyroomacoustics59```60 61Parameters62------------63 64* Data simulator parameters are contained in `conf/data_simulator.yaml`65 66Example Session67------------68 69Example multispeaker audio session (using LibriSpeech audio samples and word alignments). RTTM and CTM output labels are highlighted.70 7172 73Running the data simulator for the LibriSpeech dataset74------------75 761. Download the LibriSpeech dataset77 78```bash79python scripts/dataset_processing/get_librispeech_data.py \80 --data_root <path to download LibriSpeech dataset to> \81 --data_sets ALL82```83 842. Download LibriSpeech alignments from [here](https://drive.google.com/file/d/1WYfgr31T-PPwMcxuAq09XZfHQO5Mw8fE/view?usp=sharing) (the base directory is the LibriSpeech-Alignments directory)85 863. Create the manifest file with alignments87 88```bash89python <NeMo base path>/scripts/speaker_tasks/create_alignment_manifest.py \90 --input_manifest_filepath <Path to train_clean_100.json manifest file> \91 --base_alignment_path <Path to LibriSpeech_Alignments directory> \92 --output_manifest_filepath train-clean-100-align.json \93 --ctm_output_directory ./ctm_out \94 --libri_dataset_split train-clean-10095```96 974. (Optional) Create the background noise manifest file98 99```bash100python <NeMo base path>/scripts/speaker_tasks/pathfiles_to_diarize_manifest.py \101--paths2audio_files <Path to noise list file> \102--manifest_filepath bg_noise.json103```104 1055. Create audio sessions (near field)106 107```bash108python multispeaker_simulator.py --config-path='conf' --config-name='data_simulator.yaml' \109 data_simulator.random_seed=42 \110 data_simulator.manifest_filepath=./train-clean-100-align.json \111 data_simulator.outputs.output_dir=./test \112 data_simulator.background_noise.add_bg=True \113 data_simulator.background_noise.background_manifest=./bg_noise.json114```115 1166. Create multi-microphone audio sessions (with synthetic RIR generation)117 118```bash119python multispeaker_simulator.py --config-path='conf' --config-name='data_simulator.yaml' \120 data_simulator.random_seed=42 \121 data_simulator.manifest_filepath=./train-clean-100-align.json \122 data_simulator.outputs.output_dir=./test_rir \123 data_simulator.background_noise.add_bg=True \124 data_simulator.background_noise.background_manifest=./bg_noise.json125 data_simulator.rir_generation.use_rir=True126```127