mlx-community/mlx-model-explorer-data

MLX Model Explorer Data

An anonymous record of how people use MLX Model Explorer to choose an MLX model for their Mac: which model families, sizes, quantizations, memory classes and context lengths they look at, and which models they go on to open, compare or download. It also holds the community reports ("it worked", "too slow") and real MLX benchmark results that people choose to contribute.

The goal is to answer, with data: what is the MLX community actually trying to run, and how well does it work?

Quick start

from datasets import load_dataset

df = load_dataset("mlx-community/mlx-model-explorer-data", "events", split="train").to_pandas()
df = df[df.suspicious_flags.apply(len) == 0]                      # clean rows only

# One row per visit: keep the newest summary of each session
visits = (df[df.event_type == "session"].sort_values("timestamp")
            .drop_duplicates("session_id", keep="last"))
visits.explode("quants_searched").quants_searched.value_counts(normalize=True)

# Community MLX benchmarks: median generation speed per model and chip
bench = df[df.event_type == "mlx_benchmark"]
bench.groupby(["selected_model", "chip"]).generation_tps.median()

What is recorded

The dataset records outcomes, not clicks. Individual filter changes and page interactions are never logged.

Layout and updates

data/incoming/YYYY/MM/DD/<instance>-<UTC time>-<suffix>.parquet   # written by the Space, at most hourly
data/events/YYYY-MM.parquet                                       # one file per finished month
eval/wikitext-2-raw-v1-test-v1.txt                                # pinned text for --quality

The Space buffers rows and writes a new incoming file at most once an hour, or sooner after 2,000 rows. Early each month the previous month is compacted: all of its incoming files are merged into data/events/YYYY-MM.parquet, re-sent session rows are reduced to the newest one per visit, and the incoming files are deleted, all in one commit. Row values are never edited. The events config reads both locations, so recent data is always included. Every file has the same columns in the same order.

Schema (schema_version 2)

Every row has every column. Columns that don't apply to an event type are null.

suspicious_flags

Values outside hard bounds (for example negative speeds, unknown enums or oversized payloads) are rejected and never stored. Flagged rows are kept so filters stay transparent, but the Space excludes them from recommendations and statistics.

Privacy

Never collected: names, emails, accounts, IP addresses, precise location, cookies, user-agent strings, device fingerprints, full GPU description strings, or individual clicks. Timestamps are truncated to seconds. Session IDs are random per tab and not persisted. Rate limiting uses an in-memory salted hash that rotates hourly and is never written anywhere. Browsers sending Global Privacy Control or Do Not Track send nothing unless the user explicitly opts in, and anyone can opt out on the page.

Benchmark methodology

Browser (`webgpu-1`). The quick check runs a 256×256 WGSL matrix multiplication for about 1.2 s. The full test runs matrix multiplications at 256, 512 and 1024 plus a 256 MB buffer copy, about 20 s in total. Each phase is normalized to a fixed reference, and the score is the geometric mean × 1000. It measures WebGPU in the browser, not MLX or LLM inference.

MLX (`mlxbench-2`). mlx_explorer_bench.py follows mlx_lm.benchmark: it loads the model with mlx-lm, builds a random prompt of prompt_tokens tokens (seed 0), disables EOS so exactly generation_tokens are produced, does one warm-up run, then reports medians over 3 trials, plus peak memory. If generation speed differs by more than 25% between the fastest and slowest trial (usually memory pressure), speed and TTFT are left empty and only memory and perplexity are reported. Chip and RAM class come from sysctl.

Quality (`--quality`). The script tokenizes eval/wikitext-2-raw-v1-test-v1.txt from this repo: the first ~120k characters of the wikitext-2-raw-v1 test split (Salesforce/wikitext, CC BY-SA 3.0, revision b08601e), SHA-256 5dacf0e5…c730e8. It scores the first 16,384 tokens in non-overlapping 1,024-token windows and reports exp(mean cross-entropy). If the tokenizer has a BOS token, every window starts with it. It is a coarse quality signal and doesn't measure reasoning or instruction following.

mlxbench-1 added BOS only at the very start of the text, or not at all for tokenizers that don't add it on encode (Gemma). That inflated perplexity for models with a BOS token, Gemma's into the thousands. Those rows are kept but the Space only uses perplexity from mlxbench-2. Speed and memory figures from both versions are comparable. Models without a BOS token (Qwen) give the same perplexity in both.

The first quality rows were contributed by the Space's maintainer from one M3 Max. They cover every quantization in mlx-community that declares Qwen3.5-0.8B, 2B or 4B as its base model: MLX-4bit, OptiQ-4bit and MLX-8bit. Anyone can re-run them with the script to check the numbers.

Limitations

• —Self-selected and self-reported. The people who use the Space and submit data are not a random sample of MLX users, and feedback is subjective.
• —Estimated memory classes. Browsers can't read unified memory size. hardware_source = detected uses Chrome's rounded-down deviceMemory (capped at 32 GB). Prefer hardware_confirmed = true.
• —Interest ≠ usage. Visits show what people look at, not what they deploy.
• —Lost summaries. A visit whose browser crashes before it opens a model or hides the tab sends nothing.
• —Heuristic recommendations. top_model_score depends on engine_version.

Versioning

• —schema_version changes when columns change meaning or are removed. Version 1 was a per-click log used for the first day after launch. It was replaced by this design and its rows were deleted.
• —benchmark_version and engine_version identify the exact methodology behind each score.

License

CC BY 4.0. Please cite the dataset and link back to the Space.