CoolFace
Datasetpublic

Jianshu001/arabic-conversation-final-v1.0

Arabic Conversation — Final v1.0 (post-processed) Post-processed release of the original arabic-conversation-final dataset. Only assistant messages were modified; user messages, personas, metadata, factuality verdicts, and IDs are untouched. The source file (all_shuffled.jsonl) is preserved upstream — this repo holds the cleaned variant as a separate file. What changed Six rule-based cleanup passes are applied to every assistant message. All rules are… See the full description on the dataset page: https://huggingface.co/datasets/Jianshu001/arabic-conversation-final-v1.0.

sourceHugging Facemitupdated 6mo agoView on Hugging Face
0likes12downloads
Dataset Card

Arabic Conversation — Final v1.0 (post-processed)

Post-processed release of the original arabic-conversation-final dataset. Only assistant messages were modified; user messages, personas, metadata, factuality verdicts, and IDs are untouched. The source file (all_shuffled.jsonl) is preserved upstream — this repo holds the cleaned variant as a separate file.

What changed

Six rule-based cleanup passes are applied to every assistant message. All rules are deterministic regex / string operations — no LLM rewriting, no content synthesis, no fact-level edits.

RuleWhat it does
R1Collapse runs of 3+ consecutive newlines to a single blank line
R5bTighten \n\n before a bullet / numbered list item to \n (keeps list attached to its intro paragraph)
R5Within a pure-prose paragraph (no list/code/quote), collapse internal single \n (soft wraps) to a space
R2Strip low-value affirmative openers (أكيد،, بالتأكيد،, بكل سرور،, نعم، هذا هو, هذا سؤال مهم, …)
R4Strip وعليكم السلام greetings on turn-2+ assistant messages (only appropriate on the first turn as a greeting response)
R3Strip trailing follow-up invitation sentences on the LAST assistant turn (هل تحتاج مزيد…؟, إذا احتجت…, etc.)

Rules are applied in that order. List/code/quote paragraphs are never touched by R5. Only the last assistant message in each conversation is touched by R3.

Statistics over the full dataset

metricvalue
records11,340
assistant messages45,411
messages modified18,683 (41.1%)
R1 hits2
R5b hits11,253
R5 hits3,945
R2 hits10,506
R4 hits5
R3 hits1
\n\n per changed msg7.4 → 5.8

File

  • —all_shuffled.cleaned.jsonl — JSONL, one conversation record per line.

Schema is identical to the source dataset: id, domain, domain_ar, topic, topic_ar, subtopic_ar, persona, conversation, metadata, factuality.

Versioning

  • —v1.0 — first cleaned release (rules R1/R2/R3/R4/R5/R5b).
  • —Future revisions will bump the repo version tag and keep earlier releases available for diff / ablation.