jenny0830/celarai_early_literacy_public_Llama-31-8B-Instruct

hf_configuration:
  hf_dataset_name: celarai_early_literacy_public_Llama-31-8B-Instruct
  hf_organization: $HF_ORGANISATION
  hf_token: $HF_TOKEN
  local_dataset_dir: data/early_literacy_public/Llama-3.1-8B-Instruct/dataset
  export_jsonl: true
  jsonl_export_dir: data/early_literacy_public/Llama-3.1-8B-Instruct/jsonl
  push_to_hub: true
model_list:
- model_name: meta-llama/Llama-3.1-8B-Instruct
  base_url: http://localhost:8000/v1
  api_key: $API_KEY
  max_concurrent_requests: 8
pipeline:
  ingestion:
    source_documents_dir: yourbench/materials/early_literacy_preview
    output_dir: data/early_literacy_public/Llama-3.1-8B-Instruct/pipeline
    supported_file_extensions:
    - .md
    - .txt
    - .pdf
  summarization:
    max_tokens: 16384
    token_overlap: 128
  chunking:
    l_max_tokens: 512
    token_overlap: 0
  single_hop_question_generation:
    additional_instructions: "## Question Quality Rubric\n\nFollow these rules strictly\
      \ when generating questions:\n\n### 1. Sentence Length and Complexity\n- Use\
      \ short, simple sentences with fewer than 10 words per sentence.\n- Do NOT use\
      \ complex phrases or compound sentence structures.\n\n### 2. Vocabulary\n- Use\
      \ child-friendly words (e.g., use \"found\" instead of \"discovered\", \"finished\"\
      \ instead of \"accomplished\").\n- Do NOT use complex or meta-language (e.g.,\
      \ do NOT use words like \"plot\", \"theme\", \"narrative\", \"character development\"\
      , \"symbolize\").\n\n### 3. Focus on Key Ideas and Details\n- Ask questions\
      \ about events or details that are critical to understanding the story or that\
      \ drive the story forward.\n- Do NOT ask about tangential facts or minor details\
      \ that are unessential to the story.\n- If asking about feelings, focus on critical\
      \ moments in the story.\n\n### 4. Difficulty Level\n- Target difficulty level:\
      \ 0-2 out of 10.\n- Questions should be appropriate for 1st-2nd grade readers.\n\
      - Literal questions should be directly answerable from the text.\n- Inferential\
      \ questions should require only simple reasoning.\n- Connection questions should\
      \ relate to everyday childhood experiences.\n\n### 5. Text Structure\n- Before\
      \ generating questions, identify whether the text is narrative or informational.\n\
      - Narrative text: Focus questions on characters, setting, problem, series of\
      \ events, or solution.\n- Informational text: Focus questions on main ideas\
      \ and key supporting details.\n\n### 6. Question Types and Requirements\n\n\
      Generate exactly 3 questions per type (15 questions total). Label each question\
      \ with its type in parentheses at the end.\n\n**Literal** \u2014 The answer\
      \ is directly and explicitly stated in the text. No reasoning or background\
      \ knowledge needed.\n\n**Coherence Inferential** \u2014 The student connects\
      \ or compares information from different parts of the text to answer.\n\n**Knowledge-Based\
      \ Inferential** \u2014 The student uses a specific detail from the text together\
      \ with background knowledge to answer.\n\n**Elaborative Inferential** \u2014\
      \ The student uses background knowledge to provide information not explicitly\
      \ stated in the text.\n\n**Connection** \u2014 The student relates a specific\
      \ story theme or event to their own personal experience. Do NOT ask generic\
      \ personal questions unrelated to the story.\n\n- Do NOT repeat the same or\
      \ similar questions within a set.\n- All 15 questions must be distinct and cover\
      \ different aspects of the text.\n\n---\n\nHere are some example materials and\
      \ questions. Pay attention to the difficulty levels.\n\n# 1st Grade Texts\n\
      ## The Cat and the Map\nJan had a cat.\nThe cat was Sam.\nSam sat on Jan\u2019\
      s lap.\nJan had a map.\n\u201CIt is a map of the park,\u201D said Jan.\n\u201C\
      I can run and hop at the park!\u201D\nSam ran to the map.\nSam sat on it.\n\u201C\
      Oh, Sam!\u201D said Jan. \u201CThat is my map!\u201D\nJan got the map.\nSam\
      \ ran to the mat.\nJan ran to Sam.\n\u201CLet\u2019s go, Sam!\u201D said Jan.\n\
      Jan and Sam ran and ran.\nThey ran up a path.\nThey ran past a big rock.\nAt\
      \ last, they sat and had a nap.\nJan had fun.\nSam had fun.\n\n\nQuestions:\
      \ \nWhat was the name of Jan\u2019s cat? (literal)\nHow do you think Jan felt\
      \ when Sam sat on the map? (inferential)\nJan said she likes to run and hop\
      \ at the park. What are your favorite things to do at the park? (connection)\n\
      \n\n\n## Jobs on the Farm\nA farm has lots of jobs.\nFarm hands can work on\
      \ a farm.\nA dog can help, too!\nRon can chop logs.\nHe can mop the shop.\n\
      He can fix a box or a rod.\nDot can drop corn for the hogs.\nThe hogs trot to\
      \ the food.\nThe hogs do not stop!\nSol can jog to the pond.\nShe can hop on\
      \ a rock.\nShe can spot a frog or a log.\nJobs on a farm do not stop.\nA farm\
      \ has lots to do!\n\n\nQuestions: \nWhat are two jobs from the text that can\
      \ be done at the farm? (literal)\nWhy are there so many jobs to do on a farm?\
      \ (inferential)\nWhat are some jobs that you help with? These can be jobs at\
      \ home, at school, or anywhere you go. (connection)\n\n\n\n# 2nd Grade Texts\n\
      ## The Lost Stamp\nStan had a red stamp on his desk.\n He did not use the stamp\
      \ a lot, but he liked it.\n One day, Stan went to get the stamp.\n The stamp\
      \ was not on the desk!\nStan ran to Star.\n \u201CStar, I lost my red stamp.\
      \ Can you help me?\u201D\n \u201CYes,\u201D said Star. \u201CLet us stop and\
      \ think. Did you set the stamp on a stand?\u201D\nStan did stop and think.\n\
      \ \u201CI did set the stamp on the big stand in class,\u201D he said.\n Star\
      \ and Stan went to the stand.\n The stamp was not there.\nNext, Star and Stan\
      \ went to the steps.\n \u201CDid the stamp fall on the steps?\u201D Star said.\n\
      \ Stan bent to look.\nThere was no stamp on the steps.\nAt last, Stan went to\
      \ the back desk.\n The stamp was stuck to a stack of pads!\n Stan held up the\
      \ stamp.\n \u201CHere it is!\u201D he said.\nStar and Stan clap.\n \u201CNext\
      \ time, I will not lose my stamp,\u201D Stan said.\n Star and Stan went back\
      \ to class with a big grin.\n\n\nQuestions: \nWhat did Stan lose? (literal)\n\
      How did Stan and Star find the stamp? (inferential)\nWhere did the story take\
      \ place? How do you know? (inferential)\nDescribe a time when you lost something\
      \ important and how you found it. (connection)\n\n\n\n## Fish in the Pond\n\
      Fish can swim fast in a pond. A fish will swish its fins to dash and splash.\
      \ Some fish can rush to get food. A fish can snap up a bug or a small bit of\
      \ mash.\nFish can hide in mud or in a big bush by the pond. This helps the fish\
      \ when a crab or a bird rushes in.\nA fish has a soft body and the skin has\
      \ fins. The fins can flash in the sun.\nSome fish live in fresh ponds. Some\
      \ fish live in the big sea. Fish can be big or small. Fish can be red or tan.\
      \ All fish can swim and splash!\n\n\nQuestions: \nWhat are two things you learned\
      \ about fish from this book? (literal)\nWhy do fish hide in the mud or bushes?\
      \ (inferential)\nFish can live in fresh ponds or the ocean. What other animals\
      \ can you think of that also live in ponds or oceans? (connection)\n\n\n\n##\
      \ The Missing Ring\nMing was digging in the sand. She was hoping to find a shell.\n\
      As she was digging, she saw a ring. The ring was shining in the sun.\nMing went\
      \ running to Dad.\n\u201CLook! I found a ring!\u201D she said.\nDad was smiling.\
      \ \u201CThat is Mom\u2019s ring! She was missing it.\u201D\nMom came rushing\
      \ from the deck.\n\u201CMy ring! Thank you for bringing it to me, Ming!\u201D\
      \nMom gave Ming a big hug.\nMing kept smiling. Finding the ring was the best\
      \ part of the day.\n\n\nQuestions: \nWhere did Ming find the ring? (literal)\n\
      Why was finding the ring the best part of the day? (inferential)\nTalk about\
      \ a time that you found something. (connection)\n"
    chunk_sampling:
      enable: false
      num_samples: 100
      strategy: random
      random_seed: 42
  prepare_lighteval:
    single_hop_subset: single_hop_questions
    multi_hop_subset: multi_hop_questions
    cross_doc_subset: cross_document_questions
    chunked_subset: chunked
    summarized_subset: summarized
    output_subset: prepared_lighteval
  citation_score_filtering: {}