tonywu71/neomme-retriever-demo

NeoMME-Retriever

  

Upload PDFs or images, index them with NeoMME, then ask a question and read an answer taken from the pages that ranked highest.

How it works

The app converts every PDF page to an image because NeoMME reads pixels, not extracted text. NeoMME turns each page into token vectors and compares the query with those vectors using MaxSim. The app runs that scoring itself, so it does not need a separate index server.

Choose NeoMME 260M or 800M before indexing. Changing the model clears the index because vectors from different models cannot be compared.

The app also stores one pooled vector for each page. You can switch to dense scoring in the retrieval settings, but MaxSim is the default and the only mode covered by the published retrieval results.

When you ask a question, the app retrieves the highest-ranked pages first. It then sends the question and those page images to a vision language model, which writes an answer using the retrieved pages as context. The Space includes a small local model, and you can also use OpenAI, Anthropic, or Gemini. The app uses a visitor's key for one request and does not store it.

The app uses NeoMMEForRetrieval and NeoMMEProcessor from transformers. It does not depend on the research neomme package.

<!-- Image placeholder: visual RAG flow from user question, to top-k retrieved page images, to the vision language model, to the answer. -->

Deploying

GitHub holds the source of truth, not HuggingFace Hub. Every push to main runs .github/workflows/sync-to-hub.yml, which copies the repo to the Space, and you can also start it by hand from the Actions tab. Pushes that only touch .github are skipped because the action never uploads that directory.

The workflow needs a GitHub Actions secret named HF_TOKEN, which is a write token for the tonywu71 account. It is not the same token as the Space secret of the same name, which is a read token for the Hcompany org.

To deploy without the workflow, for example while debugging:

hf upload tonywu71/neomme-retriever-demo . --repo-type space \
  --exclude "**/__pycache__/**" --exclude ".git/**" --exclude ".github/**"

The copy only goes from GitHub to the Hf Space, so anything you edit in the Space web UI is lost on the next deploy.

Running locally

Use a separate virtual environment, because these requirements leave torch unpinned and would conflict with the research repo's pinned versions.

uv venv --python 3.12 .venv-space
VIRTUAL_ENV=.venv-space uv pip install -r requirements-local.txt
.venv-space/bin/python app.py                     # http://127.0.0.1:7860
.venv-space/bin/python smoke_test.py              # offline check with a tiny random model

requirements.txt is the build file for the Space, and it leaves out gradio and spaces because the Spaces runtime installs both and pinning them breaks ZeroGPU. requirements-local.txt adds them back for local runs.

HF_TOKEN has to be a token that can read both private checkpoints. The app loads both sizes once so different browser sessions can use different models safely. On Apple silicon it selects MPS and bfloat16. Only CPU falls back to float32.

If the answer model fails to load, the app still starts and offers only the providers that need a key.

<details> <summary>Environment variables</summary>

</details>

<details> <summary>Space settings</summary>

These settings live on the Space rather than in git, so a rebuild from scratch does not restore them.

Hardware cannot be set from this repo. The hardware: key in the front matter above is ignored, so use the Space settings page.

</details>

Citation

@misc{lac2026neommesingletowermultimodalnativemultilingual,
      title={NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference},
      author={Aurélien Lac and Tony Wu},
      year={2026},
      eprint={2609.01657},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2609.01657},
}