NaverHustQA/TVPL

TVPL (thuvienphapluat.vn)

1. 1.structured_data_doc.parquet: preprocessed version from only needed doc from tvpl . Please read by Datasets library
2. 2.parent_nodes.parquet: parent nodes from [1] by chunking with SentenceSplitter, chunk_overlap=0, chunk_size=800, tokenizer="Viet-Mistral/Vistral-7B-Chat"
3. 3.child_nodes.parquet: child nodes from [2] by chunking with SentenceSplitter, chunk_overlap=30, chunk_size=190 and using Word Segmentation, vietnamese-bi-encoder

Dedup

WARNING: IF YOU TRAIN PLEASE ONLY CONSIDER dedup/filtered_corpus.parquet and dedup/newtraintestdivide/filtered_corpus.parquet

1. 1.filtered_corpus.parquet: Merged dataset [SFT-Law]+[TVPL-structured]+[Zalo-corpus]. Clustered and then filtered. Items with the longest text are kept.
2. 2.text: str
3. 3.oid: Item's unique ID among 3 merged dataset, int
4. 4.__cluster__: int
5. 5.dataset: Original dataset where the item is extracted from, str
6. 6.__cluster_member__: oid of cluster members from the original 3.

eg:

{
  "text": "Thông tư này hướng dẫn tuần...",
  "dataset": "zalo_legal_corpus",
  "oid": 0,
  "__cluster__": 0
}

1. 1.newtraintestdivide folder contains newly divided SFTlaw with 10k tests. `oid` and `cluster` has the same meaning as in `filteredcorpus.parquet`.

1. 1.data_remapped/{file_name}.parquet: data files taken from other repoes, added 2 fields: oid (Item's unique ID among 3 merged dataset, int) and __cluster__ (Cluster ID, int).
2. 2.sfttest+sfttrain taken from [SFT-Law]

    {
    "reference": [
        "https://thuvienphapluat.vn/..."
    ],
    "answer": "Sinh viên học nghệ thuật ca trù ....",
    "question": "Sinh viên học nghệ thuật ca trù tại ...?",
    "domain": ["Tài chính nhà nước"],
    "text": "Căn cứ tại khoản 1 Điều 3 T....",
    "oid": 61425,
    "__cluster__": 219110
    }

• —tvpldataset taken from [TVPL-structured] (only the `structureddata_doc.parquet` file)

    {
    "meta_data": {
        "base": "Căn cứ Luật Tổ chức Chính phủ ngày 19 tháng 6 năm ...",
        "content": "QUY ĐỊNH XỬ PHẠT VI PHẠM HÀNH CHÍNH...",
        "date": "31/12/2021",
        "department": "Chính phủ",
        "doc_type": "Nghị định",
        "file_name": "Decree_No._139_2021_ND-CP_dated_December_31,_2021_.json",
        "id_doc": "139/2021/ND-CP",
        "location": "Hà Nội",
        "title": "Decree No. 139/2021/ND-CP dated December 31, 2021 on Administrative penalties for inland waterway navigation violations",
        "updated": 1710417820
    },
    "child_data": [
        {
        "__cluster__": 253403,
        "header": [
            "Decree No. 139/2021/ND-CP dated...",
            "Chương I. NHỮNG QUY ĐỊNH CHUNG"
        ],
        "len_tokenizer": 157,
        "lower_segmented_text": "điều 1 . phạm_vi điều_chỉnh...",
        "oid": 232323,
        "pointer_link": ["Chương I", "Điều 1"],
        "text": "Điều 1. Phạm vi điều chỉnh\n1. ..."
        }
        ]
    }

• —zalolegalcorpus from [Zalo-corpus] (only the legal_corpus.json file). law_id is append to articles.

    {
    "article_id": "1",
    "text": "Thông tư này hướng dẫn tuần tra,...",
    "title": "Điều 1. Phạm vi áp dụng",
    "law_id": "01/2009/tt-bnn",
    "oid": 0,
    "__cluster__": 0
    }

Please refer to the corresponding repositories for document on the data.

[SFT-Law]: https://huggingface.co/datasets/bkai-foundation-models/SFT-Law [TVPL-structured]:https://huggingface.co/datasets/bkai-foundation-models/TVPL/blob/main/structureddatadoc.parquet [Zalo-corpus]: https://huggingface.co/datasets/bkai-foundation-models/zalolegal2021/blob/main/original/legal_corpus.json