CoolFace
Apppublic

mteb/leaderboard-analytics-service

sourceHugging Faceupdated 5mo agoView on Hugging Face
1likes
App README

Leaderboard Analytics Metrics Spec

This project analyzes user behavior on the MTEB leaderboard page from event logs in MongoDB.

The primary purpose of this document is to define what is measured, where each metric comes from, and how each metric is calculated.

Data Contract

All analytics are based on the events collection and the following stable fields:

  • —Core dimensions: event_name, timestamp, session_id
  • —Behavior context: benchmark, filters
  • —Visitor identity (approximate): properties.visitor_id
  • —Change context: properties.old_value, properties.new_value, properties.filter_name

Important event names:

  • —page_view
  • —benchmark_change
  • —filter_change_* (dynamic names, such as filter_change_task_type)
  • —table_download (currently may be missing in some deployments)

Metrics Dictionary

1) PV (Page Views)

  • —Definition: Number of page view events.
  • —Source fields: event_name
  • —Calculation:
  • —Filter events where event_name == "page_view"
  • —PV = count of matched events

2) Sessions

  • —Definition: Number of unique interaction sessions.
  • —Source fields: session_id
  • —Calculation:
  • —Sessions = count of distinct session_id in the selected time range

3) UV (Unique Visitors, Approximate)

  • —Definition: Number of unique visitors identified by hashed fingerprint.
  • —Source fields: properties.visitor_id
  • —Calculation:
  • —Remove null/empty properties.visitor_id
  • —UV = count of distinct properties.visitor_id

4) Sessions Per Visitor

  • —Definition: Average number of sessions per visitor.
  • —Source fields: derived from Sessions and UV
  • —Calculation:
  • —Sessions Per Visitor = Sessions / UV
  • —If UV is 0, result is 0

5) Session Depth (Events Per Session)

  • —Definition: Average interaction intensity per session.
  • —Source fields: all events, session_id
  • —Calculation:
  • —Total Events = count of all events in range
  • —Session Depth = Total Events / Sessions
  • —If Sessions is 0, result is 0

Behavior Metrics

6) Benchmark Popularity

  • —Definition: Frequency of selected benchmarks.
  • —Source fields: event_name, properties.new_value
  • —Calculation:
  • —Filter event_name == "benchmark_change"
  • —Group by properties.new_value
  • —Popularity = event count per benchmark value

7) Filter Usage Distribution

  • —Definition: Usage volume by filter event type.
  • —Source fields: event_name
  • —Calculation:
  • —Filter event_name matching regex ^filter_change_
  • —Group by event_name
  • —Distribution = count per filter event

8) Filter Session Coverage

  • —Definition: Number of sessions that used each filter type.
  • —Source fields: event_name, session_id
  • —Calculation:
  • —For each filter_change_* event type:
  • —collect distinct session_id
  • —coverage = distinct session count

Funnel Metrics

Recommended session-level funnel:

  1. 1.page_view
  2. 2.benchmark_change
  3. 3.filter_change_*
  4. 4.table_download

9) Step Session Count

  • —Definition: Number of sessions that reached each funnel step.
  • —Source fields: session_id, event_name
  • —Calculation:
  • —Group events by session_id
  • —For each session, mark whether each step exists
  • —Count sessions satisfying each cumulative step condition

10) Step Conversion Rate

  • —Definition: Conversion from funnel step 1 (page_view) to each step.
  • —Source fields: derived from Step Session Count
  • —Calculation:
  • —Conversion Rate(step N) = StepN Sessions / Step1 Sessions * 100%
  • —If Step1 Sessions is 0, result is 0%

Visitor Segmentation Metrics

11) New Visitors

  • —Definition: Visitors whose current period contains their first observed visit date.
  • —Source fields: event_name, timestamp, properties.visitor_id
  • —Calculation:
  • —Use page_view events only
  • —For each visitor_id, find earliest timestamp (first_seen)
  • —If event date equals first_seen date, classify as new
  • —Count distinct visitor_id by period

12) Returning Visitors

  • —Definition: Visitors seen after their first observed date.
  • —Source fields: same as New Visitors
  • —Calculation:
  • —Use same first-seen logic
  • —If event date is later than first-seen date, classify as returning
  • —Count distinct visitor_id by period

Time Aggregation Rules

All trend metrics support these granularities:

  • —day -> %Y-%m-%d
  • —week -> %G-W%V (ISO week)
  • —month -> %Y-%m

Time filtering is applied on converted event time:

  • —Convert timestamp to datetime (ts)
  • —Keep records where start_time <= ts <= end_time

Optional benchmark filtering:

  • —If benchmark filter is provided, add benchmark == <value> to match conditions

Data Quality Notes

  1. 1.visitor_id is an approximate identifier, not a strict user identity.
  2. 2.For filter_change_*, properties.new_value may not always represent the actual final filter value; prefer filters snapshot for behavioral context.
  3. 3.If table_download is not instrumented, funnel step 4 will under-report by design.

Minimal Runtime Notes

Only required runtime inputs:

  • —MongoDB connection URI (MONGO_URI)
  • —Mongo database/collection names (defaults supported)

Local commands:

bash
uv sync
uv run leaderboard-analytics