Bendang/Informal-Standard-English-Corpus

Dataset Description

This dataset is a parallel corpus of approximately 11,000 pairs of informal conversational English text and their normalized equivalents. The informal text mimics real-world digital communication, featuring slang, phonetic spellings, missing punctuation, and abbreviations. The normalized text provides a grammatically correct and semantically equivalent version.

The dataset was created to support machine translation tasks for low-resource languages. It provides a semantic mapping of conversational intents to standard English.

Dataset Structure

The dataset contains two fields:

• —informal_english (string): Colloquial, unstructured English messages (e.g., "K do I need a login or anything", "u seen my shoes?").
• —normalized_english (string): The standardized, grammatically correct English equivalent (e.g., "Ok, do I need a login or anything?", "Have you seen my shoes?").

Data Generation Methodology

The dataset was generated using a multi-step pipeline with large language models.

1. 1.Informal text generation: The informal English text was generated using a diverse ensemble of LLMs, including ChatGPT, Claude, Z.ai, Kimi, DeepSeek, and Copilot. These models were prompted to generate highly informal, text-message-style dialogue. This approach captured a wide variety of linguistic variations, slang, and abbreviations.
2. 2.Normalization: A secondary LLM processed the informal text to generate the normalized English column. The goal was to produce an "easy English" equivalent that follows standard grammar and punctuation while preserving the original meaning.

Potential Use Cases

Most machine translation datasets rely on formal, pre-aligned corpora like parliamentary proceedings or news. Conversational parallel data of this specific nature is rare. This dataset has several applications in NLP research:

• —Low-Resource Machine Translation: For low-resource languages lacking native conversational data, this dataset acts as a semantic bridge. By translating the normalized English into a target language like Ao, researchers can train MT models to map various informal English inputs to a single standardized concept.
• —Text Normalization: The corpus can train sequence-to-sequence models to convert noisy, user-generated text into clean, standardized text. This is useful for preprocessing in search engines, virtual assistants, and chatbots.
• —Formality Style Transfer: Researchers can use this dataset to train models to adjust text formality dynamically, either translating standard English into casual slang or translating slang into formal text.
• —Data Augmentation: Intent classification models in conversational AI often struggle with informal inputs. This dataset can generate augmented training data to help classifiers map colloquial inputs to standard examples.

Limitations

• —LLM Artifacts: Because the informal text was generated by LLMs, it may contain synthetic patterns that differ slightly from genuine human SMS data.
• —Cultural Bias: The LLMs used may reflect specific biases regarding what constitutes informal text, which might limit the diversity of global English slang variations.