CoolFace
Datasetpublic

maikurokosmos/2025-jinji-awards-social-opinion

2025 Jinji Awards Social Opinion Dataset Dataset Description This dataset contains approximately 2,400 social media comments regarding key figures (e.g., Wang Baoqiang, Chen Sicheng, Song Jia, Yong Mei) and events surrounding the 2025 Jinji Awards (Golden Rooster Awards). The dataset is designed for multi-dimensional social opinion analysis, capturing Stance, Emotion, and Moral Foundations. It utilizes a "Human-in-the-Loop" annotation strategy, where initial… See the full description on the dataset page: https://huggingface.co/datasets/maikurokosmos/2025-jinji-awards-social-opinion.

sourceHugging Facemitupdated 8mo agoView on Hugging Face
1likes22downloads
Dataset Card

2025 Jinji Awards Social Opinion Dataset

Dataset Description

This dataset contains approximately 2,400 social media comments regarding key figures (e.g., Wang Baoqiang, Chen Sicheng, Song Jia, Yong Mei) and events surrounding the 2025 Jinji Awards (Golden Rooster Awards).

The dataset is designed for multi-dimensional social opinion analysis, capturing Stance, Emotion, and Moral Foundations. It utilizes a "Human-in-the-Loop" annotation strategy, where initial labels were generated by an LLM and a subset of "high-risk" samples underwent manual verification and correction.

  • —Language: Chinese (zh-CN)
  • —Total Samples: ~2,400
  • —Key Figures: Wang Baoqiang, Chen Sicheng, Song Jia, Yong Mei, etc.
  • —Annotation Types: Validity Check, Stance Detection, Emotion Classification, Moral Foundations Theory (MFT).

Dataset Structure

The dataset is stored in Parquet format. To facilitate direct analysis and visualization in the Hugging Face Viewer, nested label structures have been flattened.

Data Fields

FieldTypeDescription
idstringUnique identifier for the comment.
contentstringThe raw text of the social media comment.
is_manually_correctedboolIndicates whether this specific sample was reviewed and corrected by a human annotator.

Label Fields (Flattened)

For each annotation task, the dataset provides both the Original (AI) and Corrected (Human-verified) versions.

  • —Prefix `original_`: The initial prediction generated by the LLM.
  • —Prefix `corrected_`: The final ground truth used for analysis.
TaskField Name (Example)Description
Validitycorrected_validitygeneral (valid), noise (irrelevant), stanning_and_other_spamming (spam).
Stancecorrected_stancesupport, oppose, neutral, no_clear_stance.
Emotioncorrected_emotion_categoryanger, happiness, like, disgust, etc.
Moral Foundationscorrected_mf_maincare_vs_harm, fairness_vs_cheating, authority_vs_subversion, etc.

Creation & Methodology

This dataset was constructed using a Human-in-the-Loop (HITL) pipeline:

  1. 1.AI Annotation: An LLM provided initial labels for all samples.
  2. 2.Heuristic Filtering: Logical rules identified "high-risk" samples (e.g., Stance-Emotion mismatch).
  3. 3.Manual Correction: A subset of data was manually reviewed. The is_manually_corrected flag marks these entries.

Quality & Evaluation

The quality of the dataset was evaluated on a held-out validation set of 50 randomly sampled entries. The metrics below compare the AI's initial predictions against human ground truth.

1. Overall Performance Metrics

TaskAccuracyWeighted F1-ScoreAnalysis
Validity Check0.740.75The model effectively filters spam, achieving high recall on valid content.
Stance Detection0.710.72Reliable for explicit opinions; subtle sarcasm remains the main source of error.
Emotion Category0.510.51Lower agreement due to the subjective granularity of emotion tags (e.g., Like vs. Admiration).
Moral Foundations0.590.56Abstract moral concepts are challenging; however, Fairness detection performed reasonably well.

2. Detailed Breakdown by Class

The model performs significantly better on dominant classes. Below is the F1-score breakdown for key categories:

  • —Validity - General: 0.83 (High reliability)
  • —Stance - Support: 0.85 (Very high reliability)
  • —Stance - Oppose: 0.62 (Moderate; often confused with sarcasm)
  • —Moral - Fairness vs. Cheating: 0.57 (Captures discussions on award fairness)
  • —Moral - None: 0.78 (Reliable identification of non-moral content)

Note: Detailed classification matrices are available in the repository files (`classification_matrices_summary.csv`).

Usage

You can load this dataset directly using the Hugging Face datasets library. The Parquet format ensures fast loading.

python
from datasets import load_dataset

dataset = load_dataset("your-username/2025-jinji-awards-social-opinion-flattened")

# Example: Filter for manually corrected entries
verified_data = dataset['train'].filter(lambda x: x['is_manually_corrected'])