llamatokenizer
pile-LlamaTokenizerFast-32k-truncated-toy
LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs
📖 Paper • 🤗 HF Repo
🔍 Table of Contents
🌐 Overview
📚 Preparation
⏳ Data Selection
📈 Training
📝 Citation
🌐 Overview
Long-context modeling has drawn more and more attention in the area of Large Language Models (LLMs). Continual training with long-context data becomes the de-facto method to equip LLMs with the ability to process long inputs. However… See the full description on the dataset page: https://huggingface.co/datasets/UltraRonin/pile-LlamaTokenizerFast-32k-truncated-toy.pile-LlamaTokenizerFast-32k