Younggooo/kitrec-val-seta

KitREC Validation Dataset - Set A

Validation dataset for the KitREC (Knowledge-Instruction Transfer for Recommendation) cross-domain recommendation system.

Dataset Description

This validation dataset is designed for evaluating fine-tuned LLMs on cross-domain recommendation tasks during training. It uses the same users as the test set but allows for validation monitoring.

Dataset Summary

Set A vs Set B

• —Set A (Hybrid): Contains hard negative candidates + random candidates for challenging evaluation
• —Set B (Random): Contains only random candidates for fair baseline comparison

Both sets use the same ground truth items but differ in candidate composition.

User Type Distribution

User Type Definitions

Dataset Structure

Data Fields

• —instruction (string): The recommendation prompt including user history
• —input (string): Candidate items for recommendation (100 items per sample)
• —gt_item_id (string): Ground truth item ID
• —gt_title (string): Ground truth item title
• —gt_rating (float): User's actual rating for the ground truth item (1-5 scale)
• —user_id (string): Unique user identifier
• —user_type (string): User category (4 types)
• —candidate_set (string): A or B
• —source_domain (string): Books
• —target_domain (string): Movies & TV or Music
• —candidate_count (int): Number of candidate items (100)

Data Split

Usage

from datasets import load_dataset

# Load validation dataset
dataset = load_dataset("Younggooo/kitrec-val-seta")

# Access validation data
val_data = dataset["val"]
print(f"Validation samples: {len(val_data)}")

# Example: Filter by user type
overlapping_movies = val_data.filter(
    lambda x: x["user_type"] == "overlapping_books_movies"
)
print(f"Overlapping Movies users: {len(overlapping_movies)}")

# Example: Calculate metrics by user type
from collections import defaultdict
user_type_metrics = defaultdict(list)
for sample in val_data:
    user_type_metrics[sample["user_type"]].append(sample["gt_rating"])

Validation vs Test Data

Why Separate Validation Set?

• —Monitors training progress without data leakage
• —Enables early stopping based on validation loss
• —Validates model generalization during development

Evaluation Protocol

Metrics

• —Hit@K (K=1, 5, 10): Whether GT item is in top-K predictions
• —MRR: Mean Reciprocal Rank
• —NDCG@10: Normalized Discounted Cumulative Gain

RQ4: Confidence-Rating Alignment

Use gt_rating field to analyze correlation between model's confidence scores and actual user ratings.

Citation

@misc{kitrec2024,
  title={KitREC: Knowledge-Instruction Transfer for Cross-Domain Recommendation},
  author={KitREC Research Team},
  year={2024},
  note={Validation dataset for cross-domain recommendation evaluation}
}

License

This dataset is released under the Apache 2.0 License.