cambridge-climb/BabyLM
Dataset for the shared baby language modeling task. The goal is to train a language model from scratch on this data which represents roughly the amount of text and speech data a young child observes.
Update README.md
Update README.md
committing all txt files
Add original tagged and original gold
Add clean tagged and gold tags
Update tagging and subconfigs
Add option to load in original files
Update tagged text
Fix some errors
Strip data
Clean data again
Update BabyLM.py
Upload 3 files
Fix filepath
Fix filepath
Add tagged gold test
Add tagged test
Add tagged gold dev
Add tagged dev
Add gold tags 10M
Update loading
Add tagged 10M
Fix original files
Add rest of clean
Add cleaned 100M
Add cleaned 10M
Add scripts to tag data and improve cleaning
Clean up data a bit more
Add filename to each text row
clean-data (#1)
fixing global_idx to update on each yield
explicitly adding all filenames instead of globbing
fixing missing import
removing unspecified keyword features
v1 of dataset loading script
uploading last 2 100M files
uploading more files to 100M
Uploading partial 100M files
Adding README
uploading test data
adding dev data
Uploading strict small training data
initial commit