token-level
nepali-gector-style-token-level-tag-for-ged
Nepali GEC (gector style) Token Tagging Dataset
This is a processed version of the sumitaryal/nepali_grammatical_error_correction dataset,
designed for training GEC-ToR-style sequence tagging models.
This dataset has been processed with a robust, multi-pass, content-aware alignment algorithm
to generate high-fidelity correction tags, including complex and adjacent SWAP operations.
Total Examples: 16,260,992
Training: 13,008,711
Validation: 2,439,231
test: 813,050… See the full description on the dataset page: https://huggingface.co/datasets/DipeshChaudhary/nepali-gector-style-token-level-tag-for-ged.muril-nepali-gector-style-token-level-tag-for-ged
Nepali GEC (gector style) Token Tagging Dataset
This is a processed version of the sumitaryal/nepali_grammatical_error_correction dataset,
designed for training GEC-ToR-style sequence tagging models.
This dataset has been processed with a robust, multi-pass, content-aware alignment algorithm
to generate high-fidelity correction tags, including complex and adjacent SWAP operations.
Total Examples: 16,260,992
Training: 13,008,711
Validation: 2,439,231
test: 813,050… See the full description on the dataset page: https://huggingface.co/datasets/DipeshChaudhary/muril-nepali-gector-style-token-level-tag-for-ged.code-code-CodeCompletion-TokenLevel-Python
Dataset is imported from CodeXGLUE and pre-processed using their script.
Where to find in Semeru:
The dataset can be found at /nfs/semeru/semeru_datasets/code_xglue/code-to-code/CodeCompletion-token/dataset/py150 in Semeru
CodeXGLUE -- Code Completion (token level)
Update 2021.07.30: We update the code completion dataset with literals normalized to avoid sensitive information.
Here is the introduction and pipeline for token level code completion task.… See the full description on the dataset page: https://huggingface.co/datasets/semeru/code-code-CodeCompletion-TokenLevel-Python.wikitext-103-raw-pythia-tfidf-tokenlevelwikitext-103-raw-v4-df-tokenleveldegeneration-probe-instruct-token-level-balanced
degeneration-probe-instruct-token-level-balanced
Downsampled (1:3 positive:negative) variant of luca-sartori/degeneration-probe-instruct-token-level.
An example is considered positive if it contains at least one token with repetition >= 0.8 in the chunk_summary field. The downsampling keeps all positive examples and adds a random subset of negative examples in a 1:3 ratio.
Examples whose chunk_summary contained no scored tokens (every repetition value null) have been dropped, since… See the full description on the dataset page: https://huggingface.co/datasets/lorenzo0312/degeneration-probe-instruct-token-level-balanced.