datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Sonnet3.5-SlimOrcaDedupCleaned-4k-contextMade it fit into 4096 context length (removed 385 examples exceeding 4076 tokens on LumiOpen/Viking-7B tokenizer) and also fixed the formatting to use "human" instead of "user" due to it causing Unsloth to change "user" to "system". Original Gryphe/Sonnet3.5-SlimOrcaDedupCleaned.
ParallelFiction-Ja_En-100k-alpaca-4k-contextThis is a modified version of NilanE/ParallelFiction-Ja_En-100k which has been turned into Alpaca format.
This has also been chunked for 4096 tokens for augmxnt/shisa-base-7b-v1 model's tokenizer.
If you want the non chunked version it's here.
Dataset format (correct one)
{
'instruction' : 'Japanese chapter'
'output' : 'English translation'
'input' : 'empty'
}
Original Dataset card
Dataset details
Each entry in this dataset is a sentence-aligned… See the full description on the dataset page: https://huggingface.co/datasets/mpasila/ParallelFiction-Ja_En-100k-alpaca-4k-context.