Modelpublic

jacob-valdez/tensorcode-scene-language-experimental-001

sourceHugging Faceapache-2.0updated 3d agoView on Hugging Face

0likes

Model Card

TensorCode Scene: owned SmolVLM interpretation experiment

This is an experimental image-and-language interpretation tool, not a factual scene graph or a reliable visual reasoner. It owns the entire pretrained Idefics3 model, processor assets, and a trainable TensorCode workspace residual. No external model/provider is required after loading. Candidate ranking is a separate Scene architecture; this checkpoint uses language mode.

Inherited behavior and training

Perception and language realization are inherited from HuggingFaceTB/SmolVLM-256M-Instruct revision 7e3e67edbbed1bf9888184d9df282b700a323964. Foundation weights are frozen by default. This checkpoint has not been fine-tuned on TensorCode experience. The shared Workspace residual is randomly initialized with its output gate exactly zero. Thus the initial model preserves foundation behavior; these results establish no learned TensorCode visual-workspace contribution. Future explicit reviewer-supervised training can update the residual; the library tests only establish gradient/replay mechanics.

The owned graph uses consistent float32 weights; callers can explicitly cast the whole tool. The processor converts supplied CHW float pixels to RGB bytes before the foundation's image preprocessing. Outputs retain the input tensor's full-image source identity, shape, and SHA256 fingerprint. The model supplies no verified object boxes or extracted fact claims. Routing attention is not proof.

Measured behavior and failures

A fixed 32-photo VSR subset produced 20/32 correct caption-truth judgments (62.5%). Blank-image judgments agreed with original-image labels in 15/32 cases (46.875%); distinct-image shuffled judgments agreed in 13/32 cases (40.625%). The majority-label baseline is 17/32 (53.125%); the full model improves on it by only three examples. The controls measure original-label retention, not ground-truth accuracy for the altered images. All outputs began with a parseable yes/no. This small sample suggests image-dependent behavior but does not establish general relational reasoning.

Free descriptions included spatial composition such as a person sitting on a bed, a van in front of a building, and a cat on a suitcase. Those descriptions were not scored as factual. Blank images elicited hallucinated scenes, including invented objects and movie characters. Three of four full-image descriptions reached the 64-token limit; the API marks these outputs token_limit. Confidence is explicitly uncalibrated/unknown. Users should review interpretations against original evidence.

The images are disjoint from prior TensorCode VSR training (512 photos), verified by source IDs and exact JPEG SHA256; they also exclude the previous random-model pilot's 64 test image IDs. Foundation pretraining overlap is unknown. Dataset: cambridgeltl/vsr_random, revision b2053328fafdd018ff56cf1dfa9643caaa4e69b8. Selection: first 32 records of the previously fixed 128-image test subset. No fitting or tuning used this evaluation. Seed 23, greedy generation, 64 new tokens, remote gb10-direct GPU. See evaluation.json for all raw outputs, source receipts, and protocol. An independent offline process reproduced the first full-image judgment receipt exactly, including source fingerprint and workspace tensors; this checks persistence, not the truth of that judgment.

Use

python

from tensorcode.tools.scene import Scene

model = Scene.from_pretrained('./model')
result = model.interpret({
    'pixels': pixels,  # RGB CHW floating tensor in [0, 1]
    'question': 'Describe the arrangement and interactions in this scene.',
    'source_id': 'photo:123',
})
print(result['interpretation'])  # explicitly unverified

Install tensorcode[tools] including compatible torch/torchvision. Loading this artifact is local/offline capable and does not require loading another model. Weights and processor assets are included; original photographs, runtime sessions, and optimizer state are not.

Attribution

The SmolVLM foundation is Apache-2.0: https://huggingface.co/HuggingFaceTB/SmolVLM-256M-Instruct . VSR annotations are CC-BY-4.0: https://huggingface.co/datasets/cambridgeltl/vsr_random . Original COCO photographs retain their respective rights and are not included in this model repository.

Implementation checkpoint: b5bb0b6 in https://github.com/TensaCo/tensacode-py . Install this checkout or a later compatible release; the inherited foundation was not fine-tuned in this experiment.