CoolFace
Apppublic

obversarystudios/failure-geometry-demo

sourceHugging Facemitupdated 5mo agoView on Hugging Face
0likes
app.cpython-312.pyc98 linesDownload Raw Back to __pycache__
1�

2��i1���dZddlmZddlZddlZddlmZm	Z	ddl3mZddlm
Z
ddlmZmZddlmZmZdd	lmZmZmZee�j2d4zdzZdZd
eededeee e e e ffd�Z!dee"dejFfd�Z$dejFfd�Z%ejLdejNjQ���5Z)ejTe�ejV�5ejXdd��5ejZeej\��eej\��dd��Z/ej`dddddd �!�Z1ejdd"d#d$�%�Z3ddd�ejXd�&�5ejhd'd(d)d*�+�Z5ddd�ddd�ejV�5ejld,�-�Z7ejld.�-�Z8ddd�ejV�5ejld/�-�Z9ddd�ejV�5ejtgd0�d1d)d2�3�Z;ddd�e3jye!e/e1ge5e7e8e9e;g�4�ddd�e=d5k(re)j}�yy#1swY��xYw#1swY��xYw#1swY��xYw#1swY��xYw#1swY��xYw#1swY��xYw#1swY�ixYw)6up5Failure Geometry Demo — always runnable, no API key required.6 7Two deliberately weak baselines (always_1, keyword_heuristic) are run against8the CARB seed dataset.  Failures from both models are pooled, embedded with9TF-IDF + SVD, clustered with KMeans, and compared via mutual information.10 11The demo illustrates three questions:12 13  1. Do failure clusters align with reasoning category (transitivity, negation,14     syllogism, distractor logic)?15  2. Do they align with model identity (which baseline failed)?16  3. Visually, do the 2-D SVD projections show cluster separation?17 18No network requests, no GPU, no API token.19�)�PathN)�	BASELINES�
run_baselines)�cluster_embeddings)�load_dataset)�embed_failures�embed_for_scatter)�accuracy_by_type�compute_mi_scores)�plot_accuracy_by_type�plot_failure_scatter�plot_mi_comparison�datazcarb_demo.jsonuq## Failure Geometry Demo20 21Structured failure analysis on the CARB reasoning dataset — **no API key required**.22 23Two lightweight baselines are used intentionally:24 25| Baseline | Strategy | Expected failure shape |26|----------|----------|------------------------|27| `always_1` | Predict 1 for everything | Fails on all false-label items (systematic bias) |28| `keyword_heuristic` | Predict 0 when statement contains negation markers | Fails on affirmative-false and negated-true items |29 30Because the baselines fail *differently*, pooling their failures and clustering lets us ask:31 32> **Do clusters separate by reasoning type, by model identity, or both?**33 34Mutual information quantifies this. A larger `MI(cluster, reasoning_type)` than35`MI(cluster, model_identity)` supports the hypothesis that failure structure is36organised around reasoning difficulty — not purely around which model ran.37 38---39 40*Part of the [Obversary Studios](https://obversarystudios.org) evaluation systems research.*41*Live pipeline version (HF Inference API):*42*[carb-observability-space](https://huggingface.co/spaces/obversarystudios/carb-observability-space)*43�selected_models�44n_clusters�returnc
�z��g�dtddf�fd�}|sttj��}|dtj45���	t
t�}|dt|��dt|D�chc]}|d��	c}��d	��|d46dj|����t||��\}}|dt|��d
t|��ddt|�ztt|�d�zd�d��t|�dkr&|d�dj��dddt�fS|d�t|d��}	t|�}47|d|	j���|d|�d��t!|	|��}t#||d��D]48\}}
|
|d<�d d!lm}t)||��}|d"t+|�D�cic]}|||��49c}���|D�cgc]}|d��	}}|D�cgc]}|d#��	}}|d$�t-|||�}|j/�D]\}}|d|�d%|d&����t1|�}t3|�}t5|�}t7|50|||�}t9|�}dj��||||fS#t$r}d|��}|ddddfcYd}~Sd}~wwxYwcc}wcc}wcc}wcc}w)'N�msgrc�(���j|�y)N)�append)r�	log_liness ��]/mnt/ai_lab_os/AI_LAB/infra/09_practice/github_ready/huggingface/failure-geometry-demo/app.py�logzrun_demo.<locals>.logCs��������zLoading dataset: zDataset error: z  z examples / �reasoning_typez typeszRunning baselines: z, )�selectedu predictions · z failures (�d�z.0fz% failure rate)�z:Too few failures to cluster. Try selecting both baselines.�51u%Embedding failures (TF-IDF + SVD) …� )�n_componentsz  Embedding shape: zClustering into k=u
 (KMeans) …)rT)�strict�52cluster_idr)�Counterz	  Sizes: �model_idu Computing mutual information …z: z.4f)�str�listr�keys�	DATA_PATH�namer�	Exception�len�joinr�max�_empty_tablerr	�shaper�zip�collectionsr%�dict�sortedr�itemsr53rrr
�_build_failure_table)rrr�dataset�exc�err�r�all_results�failures�54embeddings�scatter_coords�cluster_ids�failure�cidr%�cluster_dist�k�f�reasoning_types�model_ids_list�	mi_scores�label�score�	acc_table�fig_mi�fig_acc�fig_scatter�55summary_dfrs                            @r�run_demorP=s�����I��������y�~�~�/�0���56�I�N�N�+�,�-�+��y�)��57�"�S��\�N�,�s��+Q�A�A�.>�,?�+Q�'R�&S�SY�Z�[��58�d�i�i��8�9�:�;�)�'�O�L��K���"�S��
��.�s�8�}�o�>��#�h�-��#�c�+�&6��":�:�3�?��	P�Q��8�}�q���H�I��y�y��#�T�4��|�~�E�E��/�0���r�:�J�&�x�0�N��59�j�.�.�/�0�1��60�Z�L�
�6�7�$�Z�J�G�K��H�k�$�?�$���� #����$�$����,�-�L��)�&��2F�G�Q�a��a��(�G�I�J�K�4<�=�q�q�)�*�=�O�=�-5�6��a�61�m�6�N�6��*�+�!�+���O�I�!���)�'���u��b���r�%���%�&�'�!��-�I�
�	�
*�F�#�I�.�G�&�~�{�O�Uc�d�K�%�h�/�J��9�9�Y����+�z�I�I��Y�+���u�%���D�$��d�*�*��+��,R��.H��=��6s6�J�2J)�
J.62�J3�-J8�	J&�J!�J&�!J&r=c
���t|d���D�cgc]6}|d|d|d|d|d|ddd	t|d�d	kDrd63ndzd���8}}|rtj|�St	�Scc}w)
Nc��|d|dfS)Nr$r�)r;s r�<lambda>z&_build_failure_table.<locals>.<lambda>�s���<��!�DT�BU�0V�r)�keyr$rr&�y�64prediction�x�Pu…���clusterrr&�expected�	predicted�
input_preview)r5r-�pd�	DataFramer0)r=rE�rowss   rr7r7{s�����&V�W�65�
����� 0�1��*�
��#���<���s�V�C�R�[�S��3��[�2�5E�E�2�N�
	66�67�D�68�"&�2�<�<���9�<�>�9��69s�;A/c�2�tjgd���S)Nr[)�columns)r`rarSrrr0r0�s��
�<�<�c��rzFailure Geometry Demo)�title�themeri)�scale�	min_widthzBaselines to runzOSelect one to isolate a single failure geometry; both to see model-identity MI.)�choices�valuerI�info���zKMeans clusters (k)z74 matches the number of reasoning types in the dataset.)�minimum�maximum�steprjrIrkzRun Analysis�primary�lg)�variant�size)rgzPipeline log�70Fu!Click 'Run Analysis' to start …)rI�lines�interactive�placeholderzMutual information comparison)rIz'Accuracy by reasoning type and baselinez42-D failure embedding scatter (SVD component 1 vs 2)r[u6Failure records (sorted by cluster → reasoning type)T)�headersrIrx�wrap)�fn�inputs�outputs�__main__)?�__doc__�pathlibr�gradio�gr�pandasr`�
core.baselinerr�core.clusterr�core.datasetr�71core.embedrr	�core.metricsr72r�	viz.plotsrr
r�__file__�parentr*�_DESCRIPTIONr(r'�int�tuple�objectrPr4rar7r0�Blocks�themes�Soft�demo�Markdown�Row�Column�
CheckboxGroupr)�model_selector�Slider�n_clusters_slider�Button�run_btn�Textbox�73status_log�Plot�mi_plot�acc_plot�scatter_plot�	Dataframe�
failure_table�click�__name__�launchrSrr�<module>r�s����"���2�+�%�8�<�U�U�
��N�!�!�F�*�-=�=�	���8;J��#�Y�;J��;J��3�����.�/�;J�|:�4��:�:�"�,�,�:��b�l�l���R�Y�Y�,�B�I�I�N�N�4D�E�2���B�K�K���	������
�R�Y�Y�Q�#�
.�	N�-�R�-�-��^�Y�^�^�-�.��>�9�>�>�+�,�(�f�	�N�!*��	�	�����+�N�
!�� �b�i�i��	��M�G�	N�"�R�Y�Y�Q�
�	�#����$��!�?�	�J�	�%�474�����L��"�'�'� ?�@���2�7�7�!J�K��L�75�����76��r�w�w�H�77��78�7980�����81�$����g�J���	82�
�83��M�M��� 1�2��W�h��m�L���]2�j�z���K�K�M��c	N�	N��"	�	��%���4L�L��84�85��8687�88��M2�2�s��?$K89�#J�8A.J�&J�J�J� K90�9'J&� K91�9J2�
K92�&J>�?#K93�J�J�J�J�J#	�K94�&J/	�+K95�2J;	�7K96�>K	�K97�98K