Datasetpublic

jumafernandez/d2f-turn-embeddings-personachat

Turn embeddings for Personachat (Dialog2Flow encoder) One 768-d float16 vector per utterance of Persona-Chat (Zhang et al., 2018), released within ParlAI, computed with the frozen encoder sergioburdisso/dialog2flow-joint-bert-base (SentenceTransformer recipe, convert_to_numpy, no normalization). The encoder truncates inputs at 64 tokens. If you use these embeddings, please cite the Dialog2Flow paper (Burdisso et al., EMNLP 2024) and the source corpus. Files… See the full description on the dataset page: https://huggingface.co/datasets/jumafernandez/d2f-turn-embeddings-personachat.

sourceHugging Facemitupdated 3mo agoView on Hugging Face

0likes58downloads

Dataset Card

Turn embeddings for Personachat (Dialog2Flow encoder)

One 768-d float16 vector per utterance of Persona-Chat (Zhang et al., 2018), released within ParlAI, computed with the frozen encoder sergioburdisso/dialog2flow-joint-bert-base (SentenceTransformer recipe, convert_to_numpy, no normalization). The encoder truncates inputs at 64 tokens. If you use these embeddings, please cite the Dialog2Flow paper (Burdisso et al., EMNLP 2024) and the source corpus.

Files

—personachat_e_t.f16.npy — numpy array (n_turns, 768), float16; row i is turn i.
—personachat_dialogs.slim.pkl — pandas DataFrame aligned row-by-row with the array: columns dataset, split, dialogue_id, turn_id, speaker (no utterance text).
—personachat_e_t.f16.npy.meta.json — encoding metadata.

Utterance text is not redistributed; recover it from the source corpus (https://github.com/facebookresearch/ParlAI, license mit) joining on the row order defined by the slim frame. The dataset was released by Facebook AI within the MIT-licensed ParlAI framework, with no separate dataset license; MIT is inherited from that release. Dialogues reconstructed from the truecased distribution (bavard/personachat_truecased) taking the longest history per conversation.

Load

python

import numpy as np, pandas as pd
emb = np.load("personachat_e_t.f16.npy", mmap_mode="r")
meta = pd.read_pickle("personachat_dialogs.slim.pkl")
assert len(meta) == len(emb)