build-small-hackathon/InflectionLM

1---2title: InflectionLM3emoji: 🐨4colorFrom: purple5colorTo: blue6sdk: gradio7sdk_version: 6.18.08python_version: '3.12'9app_file: app.py10pinned: false11license: mit12short_description: Finds inflection points in model outputs and displays scores13tags:14  - track:backyard15---16 17# InflectionLM: Output and Token Visualization18 19InflectionLM is a tool designed to visualize "inflection points" in Large Language Model (LLM) generation. By leveraging independent sampling, the tool generates multiple alternative output paths and identifies tokens where the model's confidence was low, highlighting these as potential points where the generation could have diverged.20 21This is built for all of my colleagues. As University lecturers, we want students to know that AI/LLMs can generate lots of different answers for a given prompt...some right and some wrong. So I built a tool that can demonstrate just that and is easy enough for anyone to use with no coding involved. InflectionLM generates multiple 22responses using top-p and top-k as well as temperature, and displays not only all the different responses, but probability score scores for the entire response as well as for each individual token. The inflections implied in the name our places in the text where a really low probability token was used, so we can see where the response 23took a turn that it might not have taken with a different flip of the coin. The user can run with greedy decoding as as well as temperatures up to 1.5 to see how that affects the variability of responses.24 25[Watch a video demo on LinkedIn!](https://www.linkedin.com/posts/mauricio-cafiero-5481259b_buildsmallhackathon-gemma4-huggingface-ugcPost-7472245848584683521-L4hx)26 27## 🌟 Features28 29- **Diverse Response Generation**: Generates 3 independent responses for every prompt using the GEMMA 4 model, utilizing Top-P (Nucleus) and Top-K sampling to maximize diversity.30- **Confidence Highlighting**: Automatically highlights tokens with a probability score below **0.6** in red, marking them as "inflection points."31- **Interactive Visualization**: 32  - A Gradio-based GUI to input prompts and view results.33  - Ability to switch between the 3 generated responses.34  - Toggleable detailed view showing exact probability scores for every token in a response.35- **Customizable Generation**: Adjustable temperature settings to control the randomness and diversity of the output.36- **UI Preferences**: Support for both light and dark modes.37 38## 🛠️ Tech Stack39 40- **Language**: Python41- **LLM**: [GEMMA 4](https://huggingface.co/google/gemma-4-31B-it) (via Hugging Face `transformers`)42- **Deep Learning Framework**: PyTorch43- **UI Framework**: Gradio44- **Compute**: Optimized for GPU usage (via `@spaces.GPU` for ZeroGPU environments)45 46## 📁 Project Structure47 48```text49InflectionLM/50├── app.py               # Gradio web application and UI logic51├── inflections_funcs.py # Core logic for model loading, generation, and scoring52├── requirements.txt     # Project dependencies53└── Stochastic_parrot.JPG # UI asset image54```55 56## 🚀 Getting Started57 58### Prerequisites59 60- Python 3.10+61- A GPU with sufficient VRAM to load GEMMA 4 (31B)62- Hugging Face account and access to the GEMMA 4 model63 64### Installation65 661. Clone the repository:67   ```bash68   git clone <repo-url>69   cd InflectionLM70   ```71 722. Install dependencies:73   ```bash74   pip install -r requirements.txt75   ```76 77### Running the Application78 79Ensure your environment variables for Hugging Face are set (e.g., `HUGGING_FACE_HUB_TOKEN`), then run:80 81```bash82python app.py83```84 85The application will start a Gradio server. Open the provided URL in your browser to begin experimenting.86 87## 📖 How it Works88 89The tool uses **Multinomial Sampling** with Top-P and Top-K filtering to generate distinct sequences. Instead of a deterministic beam search, it samples from the probability distribution to find diverse but high-quality responses. After generation, it extracts the softmax probability of each chosen token from the model's output scores.90 91Tokens with a probability $< 0.6$ are considered "unconfident." In linguistic terms, these are the "inflections"—moments where the model's internal probability distribution was flatter, meaning alternative tokens were almost as likely, leading to potential variations in the final text.92 93--94 95Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference96