robometer/rewardeval_ui
4
1o
2 ��ei�3 � @ sx d Z ddlZddlZddlmZmZ ddlmZ ddlm Z m3Z4 ddl m Z ddl
mZ ddlmZ G d d5� d6e �ZdS ) z/7Data generator for confusion matrix analysis.8� N)�Counter� defaultdict)�Tuple)�PreferenceSample�ProgressSample)�RFMBaseSampler)� rank_0_print)�SentenceTransformerc s� e Zd ZdZddef� fdd�
Zdee fdd�Zde ee ef fd d9�Z10d efd d
�Z dede fdd�Z
dd� Zdd� Z� ZS )�ConfusionMatrixSamplera� 11 Data generator that creates task-trajectory pairs for confusion matrix analysis.12 13 For each unique task, creates samples with each trajectory to analyze14 how well the model can distinguish between different tasks.15 16 If multiple data sources are present, samples N random trajectories from each data source17 and prioritizes different language instructions by randomizing the pairing order.18 � �n_trajectories_per_sourcec s� t � jd i |�� || _td�| _| j�� t| j�� �}t dt19|�� d�| j d� i | _ |D ]}| j�
|�}t�|�| j |< q/t dt20| j �� d�| j d� | `| �� | _t dt21| j�� dt22| j�� d t23| j�� d24�� d S )
a Initialize confusion matrix sampler.25 26 Args:27 n_trajectories_per_source: Number of trajectories to sample from each data source.28 If None, uses all available trajectories.29 **kwargs: Additional arguments passed to parent class.30 z'sentence-transformers/all-MiniLM-L12-v2z%Precomputing language embeddings for z
unique tasks��verbosez Precomputed z language embeddings�31Generated z& confusion matrix sample indices from z trajectories and z tasksN� )�super�__init__r r Zsentence_model�eval�list� task_indices�keysr �lenr �task_embeddings�encode�torch�tensor� _generate_all_sample_indices�sample_indices�robot_trajectories)�selfr �kwargs� unique_tasks�task� embedding�� __class__r �B/scr/aliang80/reward_fm/rfm/data/samplers/eval/confusion_matrix.pyr s 3233 34(�z ConfusionMatrixSampler.__init__�returnc 35 C s< g }t | j�� �}tdt|�� d|� �| jd� | �� \}}tdt|�� d�| jd� | �|� |�� }| j �36|� t � }|D ]-}| j | }|d } || d7 < |�
dt|��}37|D ]} |�|| | |d |38d39�� q\q?| j �40|� td t|�� d �| jd� td
t|�� �| jd� tdtt|�� ��� �| jd� |S )a( Generate all possible task-trajectory pair sample indices.41 42 If multiple data sources exist, samples N random trajectories from each data source.43 Prioritizes different video tasks first, then prioritizes different language instructions44 when creating pairs.45 �Found z unique language tasks: r
z Processing z+ trajectories for confusion matrix analysisr" � �id�frames)�traj_idx� lang_task�46video_task�47video_pathr* r z task-trajectory pairsz Video tasks sampled: z Trajectories per video task: N)r r r r r r �#_sample_trajectories_by_data_source�_print_sampling_stats�copy�
_local_random�shuffler �dataset�get�str�append�dict�sorted�items) r r Zunique_lang_tasksZsampled_trajectories�statsZshuffled_lang_tasksZvideo_task_countr, �trajr. �traj_idr- r r r& r = s> �48 4950�� 51 z3ConfusionMatrixSampler._generate_all_sample_indicesc C s� g }i t � i d�}tdd� �}| jD ] }| j| }|�dd�}|�dd�}|| | �|� qtdt|�� dt|� � �� �| j52d � |� � D �]0\}}|D ]53} | j �
|| � qMt|� � �}54| j �
|55� td56d � |�� D ��d d
� |� � D �t � d�} | jdu r�g } |� � D ] \} }
| �|
� t|
�| d | < |d | t|
�7 < q�td|� dt| �� d�| j57d � n�t| j| d �}g } dd
� |� � D �}|58�� }d}t| �|k �r(|t|�kr�d}| j �
|� || } z!t|| �}| �|� | d | d7 < |d | d7 < W n t�y |�|� |�sY �q(Y q�w |d7 }t| �|k s�td|� dt| �� d| d � d�| j59d � tdtt| d � � ��� �| j60d � | D ]}| j| }|�d t|��}|�dd�|d |< �qQ|�| � | |d |< qF||fS ) a} Sample N random trajectories from each data source, prioritizing different video tasks.61 62 When sampling N trajectories, first selects one trajectory from each unique video task,63 then repeats in round-robin fashion until N trajectories are sampled.64 65 Returns:66 Tuple of (list of sampled trajectory indices, stats dictionary)67 )� by_source�by_task� traj_to_taskc S s t t�S �N)r r r r r r& �<lambda>� s zLConfusionMatrixSampler._sample_trajectories_by_data_source.<locals>.<lambda>� data_source�unknownr" r( z data sources: r
c s s � | ]}t |�V qd S rB �r )�.0�indicesr r r& � <genexpr>� s � zMConfusionMatrixSampler._sample_trajectories_by_data_source.<locals>.<genexpr>c S � i | ] \}}|t |��qS r rF �rG r" rH r r r& �68<dictcomp>� � zNConfusionMatrixSampler._sample_trajectories_by_data_source.<locals>.<dictcomp>)�total_available�tasks_available�
tasks_sampledNrP r@ z Data source 'z
': Using all �
trajectoriesrN c S rJ r )�iterrK r r r& rL � rM r r) z ': Sampled z out of z Tasks sampled: r* rA r? )r r r r5 r6 r8 r r r r r r; r3 r4 �sum�valuesr �extend�minr2 �next�
StopIteration�popr9 r: r7 )r �sampled_indicesr< Z trajectories_by_source_and_taskr, r= rD r. Ztasks_to_indicesr" � all_tasks� source_statsZsampled_from_sourcerH Z n_to_sampleZtask_iterators� task_listZ round_idxr> r r r& r0 { s� � 6970 � �7172� 7374� � ��7576z:ConfusionMatrixSampler._sample_trajectories_by_data_sourcer< c 77 C s& | j sdS td| j d� td| j d� t|d �� �D ]\}}td|� d|� d�| j d� qtd | j d� |d78 �� D ]N\}}td |� �| j d� td |d
� �| j d� tdt|d �� �| j d� t|d �� �D ] \}}|d �|d�}td|� d|� d|� d�| j d� qkq;td| j d� dS )z�Print detailed statistics about sampled trajectories.79 80 Args:81 stats: Statistics dictionary from _sample_trajectories_by_data_source82 Nz-83=== Confusion Matrix Sampling Statistics ===r
z%84Overall trajectories per video task:r@ z z: rQ z85Per data source breakdown:r? z Data source: z Total available: rN z Tasks available: rO rP r z �/z trajectories sampledz2==================================================)r r r: r; r r6 )r r< r" �countrD r\ Z
sampled_countr r r& r1 � s& ��z,ConfusionMatrixSampler._print_sampling_stats�sample_idx_infoc C sz |d }|d }|d }|d }| j | }| j| }|d |||d�}|�� } || d< || d< | j| |d �}86t|87d88�} | S ) z=Generate a single task-trajectory sample from stored indices.r, r- r. r/ r* )r* r- r. r/ r" �text_embedding)r= �metadata)�89trajectoryN)r5 r r2 �_get_traj_from_datar ) r r` r, r- r. r/ Z90video_trajra rb Zvideo_traj_with_taskZsample_trajectory�sampler r r& � _generate_sample_from_indices s( 9192��93z4ConfusionMatrixSampler._generate_sample_from_indicesc C s94 t | j�S rB )r r )r r r r& �__len__' s 95z ConfusionMatrixSampler.__len__c C s | � | j| �S rB )rf r )r �idxr r r& � __getitem__* s z"ConfusionMatrixSampler.__getitem__)r )�__name__�96__module__� __qualname__�__doc__�intr r r9 r r r0 r1 r rf rg ri �
__classcell__r r r$ r& r97 s 98!>m r99 )rm �randomr � collectionsr r �typingr �rfm.data.dataset_typesr r �rfm.data.samplers.baser �rfm.utils.distributedr �sentence_transformersr r100 r r r r& �<module> s