Avinaash/wildchat-stratified-sample

WildChat Stratified Sample

Dataset Description

This dataset contains a stratified sample of 263 GPT-4 conversations (347 total turns) from the WildChat dataset. The sample was carefully selected to ensure balanced representation across conversation turn positions and user message lengths.

Dataset Summary

• —Total Conversations: 263
• —Total Turns/Rows: 347
• —Average Turns per Conversation: 1.32
• —Conversation Length: 1-5 turns (conversations with >5 turns excluded)
• —Message Length Range: 124-2,327 characters
• —Source Model: GPT-4 (gpt-4-0314)
• —Languages: Multilingual (primarily English)

Stratification Methodology

The dataset was created using a two-dimensional stratification strategy:

Turn Distribution (6 bins):

• —turn_1: First turn in conversation
• —turn_2: Second turn
• —turn_3: Third turn
• —turn_4-5: Fourth or fifth turn
• —turn_6-10: Sixth to tenth turn (excluded in final sample)
• —turn_11+: Eleventh turn and beyond (excluded in final sample)

Message Length (4 quartile bins):

Based on the 50th-90th percentile range (124-2,334 characters) of the original dataset:

• —short: 124-205 characters
• —medium: 206-392 characters
• —long: 393-811 characters
• —very_long: 812-2,334 characters

Strata Coverage

The sample includes 16 unique strata combinations, ensuring balanced representation across both dimensions.

Dataset Composition

Conversation Types

• —Single-turn: 185 conversations (70.3%)
• —Multi-turn: 78 conversations (29.7%)

Turn Distribution

Length Distribution

Message Length Statistics

• —Mean: 620 characters
• —Median: 392 characters
• —Std Dev: 571 characters
• —Min: 124 characters
• —Max: 2,327 characters

Data Fields

Dataset Creation

Source Dataset

The data was extracted from the WildChat dataset, which contains real-world conversations with GPT-4 collected from the wild.

Filtering Process

1. 1.Initial dataset: 193,235 turns from 72,309 conversations
2. 2.Message length filtering: Retained only messages between 50th percentile (124 chars) and 90th percentile (2,334 chars)
3. 3.Filtered dataset: 77,360 turns from 39,919 conversations
4. 4.Stratified sampling: Selected ~300 conversations using proportional stratified sampling across turn position and message length bins
5. 5.Turn limit filtering: Removed conversations with more than 5 turns
6. 6.Final dataset: 347 turns from 263 conversations

Sampling Strategy

• —Conversation-level sampling (not turn-level) to preserve complete conversations
• —Proportional allocation to maintain original distribution
• —Random seed: 42 (for reproducibility)

Intended Uses

This dataset is suitable for:

• —LLM evaluation: Testing language models on diverse conversation scenarios
• —Conversation analysis: Studying patterns in human-AI interactions
• —Quality assessment: Developing or testing conversation quality metrics

Limitations

• —Geographic bias: Reflects the geographic distribution of WildChat users
• —Language bias: Predominantly English conversations
• —Temporal limitation: Conversations from April 2023 period
• —Model-specific: All responses generated by GPT-4-0314
• —Turn limitation: Maximum 5 turns per conversation (longer conversations excluded)
• —Length filtering: Excludes very short (<124 chars) and very long (>2,334 chars) messages
• —Small sample size: 263 conversations may not capture all edge cases

Ethical Considerations

• —PII has been redacted in conversations where redacted=True
• —Toxic content flagged but not removed (check toxic field)
• —No user consent collected for research use (public conversations)
• —May contain biases present in the original WildChat dataset

Citation

If you use this dataset, please cite the original WildChat dataset:

@article{zhao2024wildchat,
  title={WildChat: 1M ChatGPT Interaction Logs in the Wild},
  author={Zhao, Wenting and Xu, Xiang and others},
  journal={arXiv preprint arXiv:2405.01470},
  year={2024}
}

License

This dataset inherits the license from the original WildChat dataset. Please refer to the WildChat dataset page for licensing information.

Dataset Card Authors

Created by the dataset curator as part of LLM evaluation research.

Contact

For questions or issues regarding this stratified sample, please open an issue in the dataset repository.