norjordAI/nor_agriculture_multi_hop_questions_bench

<img src="https://raw.githubusercontent.com/huggingface/yourbench/main/docs/assets/yourbench-badge-web.png" alt="Built with YourBench" width="200" height="32" />

Nor Agriculture Multi Hop Questions Bench

This work is related to the project in adapting LLM to answer questions about Norwegian Agriculture in Norwegian. The dataset was generated using YourBench (v0.9.0), an open-source framework for generating domain-specific benchmarks from document collections. It needs further cleaning, verification in regards to citations and validation for diversity and topics coverage. Also, it can play a role as a prove of concept for generating a domain specific evaluation dataset in Norwegian with yourbench and can be improved by prompt engineering for yourbench and fine-turning of the model to follow instructions. This version was focused on multi hop questions that find connections between different topics and test a deeper understanding. For single hop questions, check Nor Agriculture Bench. This project is currently my personal project.

Pipeline Steps

• —ingestion: Read raw source documents, convert them to normalized markdown and save for downstream steps
• —summarization: Perform hierarchical summarization: chunk-level LLM summaries followed by combine-stage reduction
• —chunking: Split texts into token-based single-hop and multi-hop chunks
• —multi_hop_question_generation: Generate multi-hop QA pairs requiring reasoning across multiple chunks
• —citation_score_filtering: Compute overlap-based citation scores and filter QA pairs accordingly

Reproducibility

To reproduce this dataset, use YourBench v0.9.0 with the following configuration:

hf_configuration:
  hf_dataset_name: nor_agriculture_multi_hop_questions_bench
  hf_organization: $HF_ORGANISATION
  hf_token: $HF_TOKEN
  local_dataset_dir: data/saved
  export_jsonl: true
  jsonl_export_dir: data/saved/jsonl
  push_to_hub: true
model_list:
- model_name: norallm/normistral-11b-thinking
  base_url: http://localhost:8000/v1
pipeline:
  ingestion:
    source_documents_dir: data/raw
    output_dir: data/processed
    upload_to_hub: false
    supported_file_extensions:
    - .md
    - .txt
    - .pdf
  summarization:
    max_tokens: 16384
    summarization_user_prompt: custom_summarization_user_prompt.md
    combine_summaries_user_prompt: custom_combine_summaries_user_prompt.md
  chunking:
    l_max_tokens: 1024
    token_overlap: 256
  multi_hop_question_generation:
    multi_hop_system_prompt: custom_multi_hop_system_prompt.md
    multi_hop_system_prompt_multi: custom_multi_hop_system_prompt_multi.md
    question_schema: schemas/single_hop_schema.py
  prepare_lighteval:
    single_hop_subset: single_hop_questions
    multi_hop_subset: multi_hop_questions
    cross_doc_subset: cross_document_questions
    chunked_subset: chunked
    summarized_subset: summarized
    output_subset: prepared_lighteval
  citation_score_filtering: {}
debug: true

(This dataset card was automatically generated by YourBench)