jackyk02/nemotron-cc-v2.1-hq-dqa-qwen3-tokens

Nemotron-CC-v2.1 / High-Quality-DQA — tokenized with the Qwen3-8B tokenizer

Question/answer pairs extracted from `nvidia/Nemotron-CC-v2.1` (High-Quality-DQA subset) and tokenized with the Qwen/Qwen3-8B tokenizer.

In the source data each row is a web document whose tail carries synthetic QA pairs marked Question: / Answer:. Here that document is split into its original prose (context) and the individual QA pairs, each tokenized separately. The Question: / Answer: marker keywords are stripped — a stored question begins at the real question text.

Layout — two tables, joined on doc_id

Context is stored once per document rather than repeated on each of its QA pairs, so nothing is duplicated. Join on doc_id to rebuild context+QA sequences.

docs/ — one row per document (7,806,965 rows)

qa_pairs/ — one row per QA pair (62,451,070 rows)

neg_pairs/ — one row per hard negative (595,312,764 rows)

Gemini-generated hard negatives (up to 10 per QA pair, from nemotron-cc-v2.1-hq-dqa-hard-negatives), tokenized under the same contract as answer_ids (tokenize(text.strip(), add_special_tokens=False)). Covers ~95.5% of qa_pairs rows; row order within a part is NOT normalized — sort or join on (doc_id, pair_idx, neg_idx).

Token counts

Tokenized with add_special_tokens=False — no BOS/EOS and no chat template applied, so you can compose your own sequence format.

Joining

Shards are aligned: docs/part_N.parquet and qa_pairs/part_N.parquet hold the same documents, so a shard joins against its own docs table rather than the whole corpus.

Note that pyarrow's Table.join cannot be used here — its Acero engine rejects list columns as join payload (Data type list<element: uint32> is not supported in join non-key field). Use a doc_id → row map instead:

import pyarrow.parquet as pq

docs  = pq.read_table("docs/part_000000.parquet")
pairs = pq.read_table("qa_pairs/part_000000.parquet")

row_of  = {d: i for i, d in enumerate(docs.column("doc_id").to_pylist())}
ctx_col = docs.column("context_ids")

for p in pairs.to_pylist():
    ctx = ctx_col[row_of[p["doc_id"]]].as_py()
    seq = ctx + p["question_ids"] + p["answer_ids"]   # separators are your call

Notes and caveats

• —Pairs per document is usually 8, but not always. Measured over one shard the distribution runs from 1 to 17 (749,613 of 749,987 documents have exactly 8). Don't assume a fixed 8.
• —110 documents (of 7,807,075) produced no parseable QA pair and are excluded from both tables, so every docs row has at least one qa_pairs partner and there are no orphans.
• —Context length: p1 254, median 764, p99 1,935 tokens.
• —Verified: doc_id unique within each docs shard, zero orphan pairs, no empty sequences, no Question:/Answer: markers leaking into stored text, all token ids within the 151,669 vocab.

Provenance

Derived from nvidia/Nemotron-CC-v2.1, a gated dataset — the original terms and license govern this derivative. See the source dataset card for its license and access conditions.