CoolFace
Datasetpublic

albertgong1/qasper

Preprocessed QASPER dataset Working doc: https://docs.google.com/document/d/1gYPhPNJ5LGttgjix1dwai8pdNcqS6PbqhsM7W0rhKNQ/edit?usp=sharing Original: Dataset: https://github.com/allenai/qasper-led-baseline Baseline repo: https://github.com/allenai/qasper-led-baseline HF: https://huggingface.co/datasets/allenai/qasper Differences of our implementation over the original implementation: We use the dataset provided at https://huggingface.co/datasets/allenai/qasper since it doesn't… See the full description on the dataset page: https://huggingface.co/datasets/albertgong1/qasper.

sourceHugging Faceupdated 2y agoView on Hugging Face
1likes67downloads
Dataset Card

Preprocessed QASPER dataset

Working doc: https://docs.google.com/document/d/1gYPhPNJ5LGttgjix1dwai8pdNcqS6PbqhsM7W0rhKNQ/edit?usp=sharing

Original:

  • —Dataset: https://github.com/allenai/qasper-led-baseline
  • —Baseline repo: https://github.com/allenai/qasper-led-baseline
  • —HF: https://huggingface.co/datasets/allenai/qasper

Differences of our implementation over the original implementation:

  1. 1.We use the dataset provided at https://huggingface.co/datasets/allenai/qasper since it doesn't require manually downloading files.
  2. 2.We remove usage of allennlp since the Python package cannot be installed anymore.
  3. 3.We add baselines to qasper/models. Currently, we have
  4. 4.QASPER (Longformer Encoder Decoder)
  5. 5.GPT-3.5-Turbo
  6. 6.TODO: RAG (with R=TF-IDF or Contriever) implemented in LangChain?
  7. 7.We replace allennlp special tokens with the special tokens of the HF transformer tokenizer:
  8. 8.paragraph separator: '</s>' -> tokenizer.sep_token
  9. 9.sequence pair start tokens: tokenizer.sequencepairstarttokens -> tokenizer.bos_token

Usage

from datasets import load_dataset

dataset = load_dataset("ag2435/qasper")