Younggooo/kitrec-test-setb

KitREC Test Dataset - Set B

Evaluation test dataset for the KitREC (Knowledge-Instruction Transfer for Recommendation) cross-domain recommendation system.

Dataset Description

This test dataset is designed for evaluating fine-tuned LLMs on cross-domain recommendation tasks across 10 different user types.

Dataset Summary

Set A vs Set B

• —Set A (Hybrid): Contains hard negative candidates + random candidates for challenging evaluation
• —Set B (Random): Contains only random candidates for fair baseline comparison

Both sets use the same ground truth items but differ in candidate composition.

User Type Distribution

User Type Definitions

Dataset Structure

Data Fields

• —instruction (string): The recommendation prompt including user history
• —input (string): Candidate items for recommendation (100 items per sample)
• —gt_item_id (string): Ground truth item ID
• —gt_title (string): Ground truth item title
• —gt_rating (float): User's actual rating for the ground truth item (1-5 scale)
• —user_id (string): Unique user identifier
• —user_type (string): User category (10 types)
• —candidate_set (string): A or B
• —source_domain (string): Books
• —target_domain (string): Movies & TV or Music
• —candidate_count (int): Number of candidate items (100)

Data Split

Usage

from datasets import load_dataset

# Load test dataset
dataset = load_dataset("Younggooo/kitrec-test-setb")

# Access test data
test_data = dataset["test"]
print(f"Test samples: {len(test_data)}")

# Example: Filter by user type
overlapping_movies = test_data.filter(
    lambda x: x["user_type"] == "overlapping_books_movies"
)
print(f"Overlapping Movies users: {len(overlapping_movies)}")

# Example: Calculate metrics by user type
from collections import defaultdict
user_type_metrics = defaultdict(list)
for sample in test_data:
    user_type_metrics[sample["user_type"]].append(sample["gt_rating"])

Evaluation Protocol

Metrics

• —Hit@K (K=1, 5, 10): Whether GT item is in top-K predictions
• —MRR: Mean Reciprocal Rank
• —NDCG@10: Normalized Discounted Cumulative Gain

Stratified Analysis

Evaluate separately for each of the 10 user types to understand model performance across different scenarios.

RQ4: Confidence-Rating Alignment

Use gt_rating field to analyze correlation between model's confidence scores and actual user ratings.

Research Questions Addressed

Citation

@misc{kitrec2024,
  title={KitREC: Knowledge-Instruction Transfer for Cross-Domain Recommendation},
  author={KitREC Research Team},
  year={2024},
  note={Test dataset for cross-domain recommendation evaluation}
}

License

This dataset is released under the Apache 2.0 License.