CoolFace
Datasetpublic

yeshpanovrustem/kaznerd

A Named Entity Recognition Dataset for Kazakh This is a modified version of the dataset provided in the LREC 2022 paper KazNERD: Kazakh Named Entity Recognition Dataset. The original repository for the paper can be found at https://github.com/IS2AI/KazNERD. Tokens denoting speech disfluencies and hesitations (parenthesised) and background noise [bracketed] were removed. A total of 2,027 duplicate sentences were removed. Statistics for training (Train), validation… See the full description on the dataset page: https://huggingface.co/datasets/yeshpanovrustem/kaznerd.

sourceHugging Facecc-by-4.0updated 1y agoView on Hugging Face
9likes118downloads
Dataset Card

A Named Entity Recognition Dataset for Kazakh

  • —This is a modified version of the dataset provided in the LREC 2022 paper *KazNERD: Kazakh Named Entity Recognition Dataset*.
  • —The original repository for the paper can be found at https://github.com/IS2AI/KazNERD.
  • —Tokens denoting speech disfluencies and hesitations (parenthesised) and background noise [bracketed] were removed.
  • —A total of 2,027 duplicate sentences were removed.

Dataset Description

Statistics for training (Train), validation (Valid), and test (Test) sets

UnitTrainValidTestTotal
Sentence88,540 (80.00%)11,067 (10.00%)11,068 (10.00%)110,675 (100%)
Token1,088,461 (80.04%)136,021 (10.00%)135,426 (9.96%)1,359,908 (100%)
NE106,148 (80.17%)13,189 (9.96%)13,072 (9.87%)132,409 (100%)

80 / 10 / 10 split

RepresentationTrainValidTestTotal
AID67,582 (79.99%)8,439 (9.99%)8,467 (10.02%)84,488 (100%)
BID19,006 (80.11%)2,380 (10.03%)2,338 (9.85%)23,724 (100%)
CID1,050 (78.89%)138 (10.37%)143 ( 10.74%)1,331 (100%)
DID633 (79.22%)82 (10.26%)84 (10.51%)799 (100%)
EID260 (81.00%)27 (8.41%)34 (10.59%)321 (100%)
FID9 (75.00%)1 (8.33%)2 (16.67%)12 (100%)
Total88,540 (80.00%)11,067 (10.00%)11,068 (10.00%)110,675 (100%)

Distribution of representations across sets

RepresentationTrainValidTestTotal
AID67,582 (76.33%)8,439 (76.25%)8,467 (76.50%)84,488 (76.34%)
BID19,006 (21.47%)2,380 (21.51%)2,338 (21.12%)23,724 (21.44%)
CID1,050 (1.19%)138 (1.25%)143 ( 1.29%)1,331 (1.20%)
DID633 (0.71%)82 (0.74%)84 (0.76%)799 (0.72%)
EID260 (0.29%)27 (0.24%)34 (0.31%)321 (0.29%)
FID9 (0.01%)1 (0.01%)2 (0.02%)12 (0.01%)
Total88,540 (100.00%)11,067 (10.00%)11,068 (10.00%)110,675 (100%)

Distribution of NEs across sets

**NE Class****Train****Valid****Test****Total**
ADAGE153 (0.14%)19 (0.14%)17 (0.13%)189 (0.14%)
ART1,533 (1.44%)155 (1.18%)161 (1.23%)1,849 (1.40%)
CARDINAL23,135 (21.8%)2,878 (21.82%)2,789 (21.34%)28,802 (21.75%)
CONTACT159 (0.15%)18 (0.14%)20 (0.15%)197 (0.15%)
DATE20,006 (18.85%)2,603 (19.74%)2,584 (19.77%)25,193 (19.03%)
DISEASE1,022 (0.96%)121 (0.92%)119 (0.91%)1,262 (0.95%)
EVENT1,331 (1.25%)154 (1.17%)154 (1.18%)1,639 (1.24%)
FACILITY1,723 (1.62%)178 (1.35%)197 (1.51%)2,098 (1.58%)
GPE13,625 (12.84%)1,656 (12.56%)1,691 (12.94%)16,972 (12.82%)
LANGUAGE350 (0.33%)47 (0.36%)41 (0.31%)438 (0.33%)
LAW419 (0.39%)56 (0.42%)55 (0.42%)530 (0.40%)
LOCATION1,736 (1.64%)210 (1.59%)208 (1.59%)2,154 (1.63%)
MISCELLANEOUS191 (0.18%)26 (0.2%)26 (0.2%)243 (0.18%)
MONEY3,652 (3.44%)455 (3.45%)427 (3.27%)4,534 (3.42%)
NON_HUMAN6 (0.01%)1 (0.01%)1 (0.01%)8 (0.01%)
NORP2,929 (2.76%)374 (2.84%)368 (2.82%)3,671 (2.77%)
ORDINAL3,054 (2.88%)385 (2.92%)382 (2.92%)3,821 (2.89%)
ORGANISATION5,956 (5.61%)753 (5.71%)718 (5.49%)7,427 (5.61%)
PERCENTAGE3,357 (3.16%)437 (3.31%)462 (3.53%)4,256 (3.21%)
PERSON9,817 (9.25%)1,175 (8.91%)1,151 (8.81%)12,143 (9.17%)
POSITION4,844 (4.56%)587 (4.45%)597 (4.57%)6,028 (4.55%)
PRODUCT586 (0.55%)73 (0.55%)75 (0.57%)734 (0.55%)
PROJECT1,681 (1.58%)209 (1.58%)206 (1.58%)2,096 (1.58%)
QUANTITY3,063 (2.89%)411 (3.12%)403 (3.08%)3,877 (2.93%)
TIME1,820 (1.71%)208 (1.58%)220 (1.68%)2,248 (1.70%)
Total106,148 (100%)13,189 (100%)13,072 (100%)132,409 (100%)