datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
nepali-gector-style-token-level-tag-for-ged
Nepali GEC (gector style) Token Tagging Dataset
This is a processed version of the sumitaryal/nepali_grammatical_error_correction dataset,
designed for training GEC-ToR-style sequence tagging models.
This dataset has been processed with a robust, multi-pass, content-aware alignment algorithm
to generate high-fidelity correction tags, including complex and adjacent SWAP operations.
Total Examples: 16,260,992
Training: 13,008,711
Validation: 2,439,231
test: 813,050… See the full description on the dataset page: https://huggingface.co/datasets/DipeshChaudhary/nepali-gector-style-token-level-tag-for-ged.muril-nepali-gector-style-token-level-tag-for-ged
Nepali GEC (gector style) Token Tagging Dataset
This is a processed version of the sumitaryal/nepali_grammatical_error_correction dataset,
designed for training GEC-ToR-style sequence tagging models.
This dataset has been processed with a robust, multi-pass, content-aware alignment algorithm
to generate high-fidelity correction tags, including complex and adjacent SWAP operations.
Total Examples: 16,260,992
Training: 13,008,711
Validation: 2,439,231
test: 813,050… See the full description on the dataset page: https://huggingface.co/datasets/DipeshChaudhary/muril-nepali-gector-style-token-level-tag-for-ged.code-code-CodeCompletion-TokenLevel-Python
Dataset is imported from CodeXGLUE and pre-processed using their script.
Where to find in Semeru:
The dataset can be found at /nfs/semeru/semeru_datasets/code_xglue/code-to-code/CodeCompletion-token/dataset/py150 in Semeru
CodeXGLUE -- Code Completion (token level)
Update 2021.07.30: We update the code completion dataset with literals normalized to avoid sensitive information.
Here is the introduction and pipeline for token level code completion task.… See the full description on the dataset page: https://huggingface.co/datasets/semeru/code-code-CodeCompletion-TokenLevel-Python.code-code-CodeCompletion-TokenLevel-Java
Dataset is imported from CodeXGLUE and pre-processed using their script.
Where to find in Semeru:
The dataset can be found at /nfs/semeru/semeru_datasets/code_xglue/code-to-code/CodeCompletion-token/dataset/javaCorpus in Semeru
CodeXGLUE -- Code Completion (token level)
Update 2021.07.30: We update the code completion dataset with literals normalized to avoid sensitive information.
Here is the introduction and pipeline for token level code completion task.… See the full description on the dataset page: https://huggingface.co/datasets/semeru/code-code-CodeCompletion-TokenLevel-Java.wikitext-103-raw-pythia-tfidf-tokenleveldegeneration-probe-instruct-token-level-balanced
degeneration-probe-instruct-token-level-balanced
Downsampled (1:3 positive:negative) variant of luca-sartori/degeneration-probe-instruct-token-level.
An example is considered positive if it contains at least one token with repetition >= 0.8 in the chunk_summary field. The downsampling keeps all positive examples and adds a random subset of negative examples in a 1:3 ratio.
Examples whose chunk_summary contained no scored tokens (every repetition value null) have been dropped, since… See the full description on the dataset page: https://huggingface.co/datasets/lorenzo0312/degeneration-probe-instruct-token-level-balanced.wikitext-103-raw-v4-df-tokenlevelwikitext-103-raw-v3-tf-idf-tokenleveldegeneration-probe-instruct-token-level
Degeneration Probe Instruct Generations (Token-Level)
Per-token degeneration variant of the instruct generations. chunk_summary is a list of {token_index, degenerating, repetition} records, where repetition is the average bigram-repetition score over all length-256 sliding windows that contain the token, and degenerating is True when that average is >= 0.8.
The rows are sourced from /Users/lucae/Desktop/degeneration/degeneration-probe/outputs/hf_datasets/instruct_concatenated/data… See the full description on the dataset page: https://huggingface.co/datasets/luca-sartori/degeneration-probe-instruct-token-level.