sahil-12kumar/IL_CMS_Tools

1---2title: IL CMS Tools3emoji: 📝4colorFrom: indigo5colorTo: blue6sdk: docker7app_port: 78608pinned: false9 10---11 12# IL CMS Tools13 14A Flask web app for the Infinity Learn content team. It runs as one process — on15the host laptop for LAN/tunnel use, or as a Docker container (Hugging Face Space)16— and bundles nine tools that all talk to the IL CMS with the SME's *own* login.17 18> **Screen support:** the UI is laid out for laptops and desktops, **1280px wide19> and up**. Phone and tablet layouts are not maintained — pages will render but20> will not lay out sensibly below 1280px. The layout rules live in21> [frontend/static/il-laptop.css](frontend/static/il-laptop.css).22 23## The tools24 25| Tool | Page | What it does |26|------|------|--------------|27| **Home** | `/dashboard` | CMS connection card + tool cards. One login here connects every tool |28| **Uploader** | `/uploader` | Parses questions from Excel / Word / Markdown / pasted text and creates them in the CMS question bank as drafts |29| **Tagger** | `/tagger` | Pushes TOC + metadata tags from a tagging Excel via the QB REST API, then runs the approval workflow |30| **AI Tagging** | `/tag` | Builds a copy-paste tagging prompt from live CMS content, validates the AI's JSON reply, and hands it to the Tagger |31| **Q-A Verification** | `/verify` | Has an AI solve questions *blind* and flags where its answer disagrees with the stored key |32| **Solutions & Keys** | `/solutions` | Has an AI write step-by-step solutions, then writes the solution and/or corrected key back to the CMS |33| **LO Classifier** | `/lo` | Classifies questions into chapter / topic / subtopic / learning outcome against the SME's own LO sheets |34| **Quiz PPT** | `/ppt` | Renders CMS questions (MathML included) onto a branded PowerPoint template |35| **Diagram Upload** | `/upload_image` | Stores a diagram in Supabase and returns a permanent public URL to paste as `[img:<url>]` |36| **Resolve TOC** | `/resolve_toc` | Resolves pasted question IDs to their live chapter / topic / subtopic names as an Excel |37| **Usage** | `/usage` | Per-user / per-tool run stats (host laptop only) |38 39---40 41## Quick start42 43```bash44pip install -r requirements.txt45playwright install chromium        # needed by Quiz PPT, TOC lookups, make_template.py46python backend/app.py47```48 49Then open `http://127.0.0.1:7860`, or share the printed Network URL with the team.50On Windows, `start_app_only.bat` does the same thing and tags the run for the51shared usage dashboard.52 53**First run** generates a `SECRET_KEY` and a random `ACCESS_PASSWORD`, writes both54to a gitignored `.env` in the repo root, and prints the password in the startup55banner. Edit `.env` to set a password the team can remember. Nothing needs to be56configured by hand before the first launch.57 58## Signing in — two separate layers59 601. **App gate** (`/login`) — the shared `ACCESS_PASSWORD`. Keeps the LAN/tunnel URL61   from being open to anyone who finds it. Set `ACCESS_PASSWORD=` (empty) to run62   with no gate at all.632. **CMS login** — each user enters their **own CMS phone + password** on the Home64   page. That single login fills the server-side credential store (Uploader, PPT,65   Resolve TOC, Diagram Upload) *and* writes the per-browser session file the66   Tagger / AI Tagging / LO Classifier use. Sessions last 20 hours.67 68The CMS token is held **server-side**, never in the browser cookie — the cookie69only carries a client id. Restarting the server therefore disconnects everyone,70and they reconnect on the Home page.71 72**Verify** and **Solutions** can additionally connect their *own* CMS account,73independent of the Home login, so a reviewer can push under a different identity74without disturbing the other tools. Each has its own Connect/Disconnect control.75 76---77 78## 1. Uploader (`/uploader`)79 801. **Upload a file** — `.xlsx`, `.docx`, `.md`, or `.zip` (markdown + images) — or81   paste question text straight into the page. Sample templates are downloadable82   on the page; the formats are documented in [samples/README.md](samples/README.md).832. **Review** — parsing runs in the background with live progress; each question84   gets a confidence flag and a list of issues (no answer key, fewer than four85   options, …). An answer key can be pasted separately and merged by question number.863. **Upload to CMS** — questions are pushed one-by-one as drafts; a results report87   with the new question IDs downloads as Excel.88 89Math (OMML, MathType OLE, LaTeX) is converted to MathML; images become embedded90base64 or Supabase URLs. Numerical questions carry their answer range through91validation into the payload.92 93`upload_md.py` is the same engine as a CLI — `--check`, `--key/--inject`,94`--upload` — for driving a question bank from the terminal.95 96## 2. Tagger (`/tagger`)97 98The run engine is the **fast QB REST API tagger** ([api_tagger.py](api_tagger.py)):99per question it resolves chapter/topic/subtopic names to the chosen exam's node IDs,100PUTs the tags + metadata, then drives the status workflow to *approved* or101*under review*. It runs sequentially — the worker-count control is legacy and no102longer parallelises. [tagger.py](tagger.py) is kept for its Excel/session helpers103and the CMS login, plus the older Playwright engine.104 105**Tagging Excel columns** (any order; only `question_id` is mandatory):106 107```108question_id | subject | chapter | topic | subtopic | grade109objective | difficulty | bloom_taxonomy | relevance110minitopic | micro_topic | concept_level | source | syllabus111author | tags | exam | year | month | date | toc112```113 114Tracking columns (`Status`, `Run Details`, `Attempts`, `Last Worker`,115`Last Updated`) are added automatically and updated row-by-row, so the downloaded116"updated Excel" doubles as the run report. Rows marked `processed` are skipped on117re-runs — after a run with failures, just run again with the Question IDs box118empty to retry only the failed rows.119 120**Steps on the page:** upload the tagging Excel → pick the exam TOC pill (NEET,121JEE Main, …) and approval mode → Run. Live log + progress; on finish you get a122failed-questions report and the updated Excel. Each user gets their own run; a123second run for the same user is refused while one is in progress.124 125**Approval mode is admin-only:** Full Approve (the workflow all the way to126*approved*) is restricted to the admin CMS phone `9939921144`; everyone else is127locked to *Under Review* and the radio is hidden. The check is enforced in128`/tagger/start` server-side, not just in the UI. `can_approve` gates the radio129in `tagger.html` / `tag.html`.130 131`python backend/make_template.py` regenerates `database/tagging_template.xlsx` by scraping the live132CMS form with Playwright, so every dropdown column carries valid in-Excel options.133`/tagger/refresh_template` does the same from the page.134 135## 3. The AI tools — `/tag`, `/verify`, `/solutions`, `/lo`136 137These four share one design and **need no API key and no paid AI account**:138 1391. The app fetches each question from the CMS and renders it *readably* —140   [cms_text.py](backend/cms_text.py) turns MathML into `$...$` LaTeX and marks images as141   `[figure]`, so the AI never sees a half-broken question.1422. It builds **one copy-paste prompt** that also carries the valid vocabulary143   (exam TOC, dropdown options, or the LO taxonomy).1443. The SME pastes it into claude.ai / ChatGPT on their own subscription and pastes145   the JSON reply back.1464. The app **hard-validates** every value against the real taxonomy before anything147   is written — a row that doesn't resolve is rejected, not guessed.148 149| Page | Prompt asks for | Written back |150|------|-----------------|--------------|151| `/tag` | chapter/topic/subtopic + metadata for each QID | a tagging Excel, handed straight to the Tagger's push + approve flow |152| `/verify` | solve every question independently | nothing — produces a MATCH / MISMATCH / UNSURE report, plus an optional adjudicator second pass |153| `/solutions` | solve + write a step-by-step solution | the solution and, when ticked, the corrected answer key |154| `/lo` | chapter/topic/subtopic/LO from the SME's sheets | appends validated rows to `database/lo_bank/LO_Question_Bank_<Subject>.xlsx` |155 156`/verify` and `/solutions` are **solve-blind**: the stored answer is withheld from157the prompt and kept server-side, so a disagreement is real signal rather than the158AI agreeing with what it was shown. Key writes are type-aware — MCQs store 1-based159option indices, numeric types store the literal value — and are re-encoded from the160*live* record at push time.161 162## 4. Quiz PPT (`/ppt`)163 164Paste QIDs, optionally upload your own `.pptx` template, and each question +165options are rendered locally with MathJax in one persistent headless page and166dropped onto the branded template slide. Runs as a background job with a live log.167[ppt_api.py](backend/ppt_api.py) also works as a standalone CLI.168 169**Chapter / Topic title slides.** Instead of a flat QID list you can paste rows170straight from a spreadsheet — `Chapter  Topic  QID  Year/Exam` (the chapter and171topic index numbers are fine to include). Each new chapter emits a **chapter172title slide** (with a `NEET PYQ` tag), each new topic a **topic title slide**,173then the question slides beneath them. On each question slide the **year** sits174in a top-left box and **chapter + topic** in a bottom-right box. A plain flat175QID list (one per line) still works and produces question slides only.176[ppt_api.parse_tabular](backend/ppt_api.py) groups the rows by chapter then177topic in first-seen order.178 179**Chapter-wise output (ZIP).** When a pasted table produces multiple chapters,180the generator instead builds **one branded `.pptx` per chapter** and delivers181them as a single ZIP download. To put one question into several chapters, just182repeat its row under each chapter — it is fetched/rendered once and reused in183every chapter deck it appears in. ([ppt_api.generate_chapter_wise](backend/ppt_api.py)184does the fan-out; the flat plain-QID mode stays a single file.)185 186## 5. Diagram Upload (`/upload_image`)187 188Uploads a PNG/JPG/GIF/WebP (≤10 MB) to the Supabase `question-images` bucket and189returns its permanent public URL. **Requires `SUPABASE_URL` + `SUPABASE_KEY`** —190without them every attempt returns "Image storage not configured". This is the one191tool that doesn't need a CMS connection.192 193## 6. Resolve TOC (`/resolve_toc`)194 195Paste up to 500 question IDs and get an Excel of their live chapter / topic /196subtopic / grade names. It probes the community API separately from the QB API,197because the community API invalidates tokens hours earlier — otherwise the page198would say "Connected" and silently resolve every name to blank.199 200**Exam TOC / TOC Chain / Chapter / Topic all come from row 1 of the question's201"Visible to students in this exams" list** — the numbered list CMS shows on the202question page, e.g. for `ILQ-1836594`:203 204```2051. CBSE Grade 7 > Mathematics > Exponents and Powers > Laws of exponents2062. CBSE Grade 8 > Mathematics > Power Play. > Laws of Exponents2073. Grade 8-AP State Board > Mathematics > Exponents and Powers(Inactive) > ...208```209 210Rows read `exam > subject > chapter > topic`, and the `TOC Chain` column211reproduces row 1 verbatim. Those rows come from the same212`get_navigation_details_for_subtopic` call the names already needed — no extra213request. Exam TOCs include board/grade TOCs, not only JEE/NEET.214 215The order is the API's own, which is the order CMS numbers them in. It is216deliberately **not** re-sorted by the question's stored `chapter_id`: that ID217often points at a later row, which is what made earlier versions report a218chapter the SME wasn't expecting. `(Inactive)` in row 3 above is part of the219chapter *name*, not a status flag — every row in this panel is student-visible,220so no row is filtered out. Subject comes from the row too, overriding the master221name (`Mathematics`, not `Maths`). The subtopic keeps its own column and is222absent from the chain, since CMS's list stops at the topic.223 224Questions with no rows at all fall back to the master TOC names and a blank225`Exam TOC`.226 227**Speed.** A resolve is two API calls per question and nothing else, so it is228pure I/O wait. Three things keep a 500-ID batch quick: `RESOLVE_TOC_WORKERS`229threads (default 12, clamped 1–32 — raise it if runs stay clean, drop it if rows230start coming back as `ERROR`), one pooled `requests.Session` so calls reuse231connections instead of paying a TLS handshake each, and a per-job cache of nav232responses keyed by subtopic. That last one matters most in practice: a pasted233batch is usually a single chapter, so a few subtopics repeat across every row.234The cache is per job, never process-wide, so a TOC edited in CMS shows up on the235next run.236 237---238 239## Repo layout240 241```242IL_CMS_Tools/243├── backend/                  # All Python code244│   ├── app.py                #   Flask app — every route for all nine tools245│   ├── tagger.py             #   CMS login, tagging-Excel helpers, legacy Playwright engine246│   ├── api_tagger.py         #   Fast QB REST tagging engine (the one /tagger runs)247│   ├── ai_tagger/            #   Shared CMS API primitives248│   │   ├── push_tags.py      #     QB headers from a state file, fetch, resolve_ids, push_one249│   │   ├── approve_tags.py   #     status workflow -> under review / approved250│   │   └── extract_toc.py    #     exam TOC name -> node id lookups (shared service account)251│   ├── ai_assist.py          #   AI Tagging bridge — prompt build + JSON validation -> Excel252│   ├── tag_assist.py         #   Curated exam TOC + merged CMS option lists for /tag253│   ├── verify_assist.py      #   Solve-blind answer verification + adjudicator pass254│   ├── solution_assist.py    #   Solution authoring + type-aware answer-key writeback255│   ├── lo_assist.py          #   LO Classifier — taxonomy validation + per-subject bank256│   ├── cms_text.py           #   CMS question HTML/MathML -> readable text for prompts257│   ├── ppt_api.py            #   Quiz PPT generator (MathJax render -> branded slides)258│   ├── make_template.py      #   Scrapes the CMS form -> database/tagging_template.xlsx259│   ├── upload_md.py          #   CLI wrapper around the uploader engine260│   ├── mathtype_mtef.py      #   MathType OLE equation streams -> MathML261│   ├── il_shared_usage.py    #   Fire-and-forget mirror of usage events to Supabase262│   ├── build_exam_toc.py     #   SME TOC sheets  -> database/exam_toc.json (+ review workbook)263│   ├── build_lo_taxonomy.py  #   SME LO sheets   -> database/lo_taxonomy.json264│   ├── build_cms_options.py  #   CMS /lookups    -> database/cms_options.json snapshot265│   └── vendor/               #   OMML2MML.XSL — Word-equation conversion (committed)266├── frontend/267│   ├── templates/            # home, uploader, tagger, tag, verify, solutions, lo,268│   │                         # ppt, upload_image, resolve_toc, usage, login, _nav269│   └── static/               # il-laptop.css (layout system), il-common.js270├── database/                 # Data the tools read/write271│   ├── exam_toc.json         #   Curated per-exam TOC, snapped to exact CMS names272│   ├── lo_taxonomy.json      #   Subject -> chapter -> topic -> subtopic -> LO273│   ├── cms_options.json      #   Fallback metadata option lists274│   ├── ppt_template.pptx     #   Default Quiz PPT template275│   ├── tagging_template.xlsx #   Generated tagging template (gitignored)276│   └── ...                   #   runtime state (gitignored) — see below277├── samples/                  # Downloadable sample question + tagging files278├── tests/                    # pytest suite (imports the app from backend/)279├── Dockerfile                # HF Space / container image280└── requirements.txt281```282 283Runtime-only, all gitignored: `.env` (repo root), `state_*.json`, and under284`database/`: `tagger_sessions/`, `tagger_reports/`, `tagger_screenshots/`,285`lo_bank/`, `usage_log.csv`.286 287### Rebuilding the data files288 289Run on the host laptop with a fresh CMS login, then commit the JSON:290 291```bash292python backend/build_exam_toc.py       # when the curated TOC sheets or the CMS tree change293python backend/build_lo_taxonomy.py    # when an LO sheet changes294python backend/build_cms_options.py    # when CMS changes the metadata dropdowns295python backend/make_template.py        # when the CMS question form changes296```297 298`build_exam_toc.py` also writes `database/exam_toc_review.xlsx` — filter `status != exact`,299fix genuine mis-snaps in `exam_toc_aliases.xlsx`, and re-run.300 301---302 303## Configuration (environment variables)304 305Every value below can live in `.env` in the repo root (auto-created, gitignored) or306in the real environment — a real environment variable always wins. Docker / HF307Space deployments supply them as Space secrets.308 309| Variable | Required | Purpose |310|----------|----------|---------|311| `SECRET_KEY` | auto | Flask session signing. Generated on first run and saved to `.env`. Changing it signs everyone out |312| `ACCESS_PASSWORD` | auto | App login gate. A random one is generated and printed on first run. Set it to an empty string to *deliberately* run with no gate |313| `SUPABASE_URL` | for Diagram Upload + usage | Supabase project URL. Without it `/upload_image` cannot upload anything, and usage events aren't mirrored to the shared dashboard |314| `SUPABASE_KEY` | for Diagram Upload + usage | Supabase key with write access to the `question-images` bucket and the `usage_events` table (`SUPABASE_SERVICE_KEY` is accepted as an alias) |315| `PORT` | no | Server port (default `7860`) |316| `HTTPS_ONLY` | no | `1` marks the session cookie Secure. Only turn this on if nobody reaches the app over plain http on the LAN — a Secure cookie is never sent over http |317| `DEV` | no | `DEV=1` enables debug mode + auto-reload on code edits |318| `IL_USAGE_SOURCE` | no | Label for this process in the shared usage dashboard (`start_app_only.bat` sets `web-app`) |319| `PPT_CRED_PHONE` / `PPT_CRED_PASSWORD` | no | Credentials for `ppt_api.py` run as a standalone CLI. The web app always uses the SME's own session instead |320| `IL_TOKEN` | no | CMS token for `upload_md.py` run as a standalone CLI |321| `IL_QB_API` | no | Question Bank REST API host. Defaults to the **production** stack (`https://newqbapi.infinitylearn.com`) |322| `IL_GATEWAY_API` | no | Login/auth gateway host. Defaults to `https://gatewayapicms.infinitylearn.com` |323| `IL_COMMUNITY_API` | no | Community API host (TOC tree, approvals). Defaults to `https://communityapi.infinitylearn.com` |324| `IL_COMMUNITY_WEB` | no | Browser-facing CMS site — what Playwright opens and what goes in `referer` headers. Defaults to `https://community.infinitylearn.com` (note: **no** `api`, unlike `IL_COMMUNITY_API`) |325 326### Dev vs production endpoints327 328Every CMS host name lives in [backend/il_endpoints.py](backend/il_endpoints.py) and329**defaults to the production stack** — with nothing set, every tool reads and writes330real CMS content. Nothing else in the code hardcodes a host, so pointing the whole app331at dev/preprod is four lines in `.env`:332 333```334IL_QB_API=https://newqbapi.devinfinitylearn.in335IL_GATEWAY_API=https://gatewayapi.devinfinitylearn.in336IL_COMMUNITY_API=https://cmspreprodapi.devinfinitylearn.in337IL_COMMUNITY_WEB=https://cmspreprod.devinfinitylearn.in338```339 340`IL_COMMUNITY_WEB` is a **different host** from `IL_COMMUNITY_API`, not a duplicate:341the API answers on `communityapi.infinitylearn.com`, while the CMS a browser loads342is `community.infinitylearn.com` (no `api`). The web value is what Playwright opens343for the Tagger and `make_template.py`, and what fills `referer` headers — pointing it344at the API host breaks both tools.345 346```powershell347# Overriding for one run (PowerShell)348$env:ACCESS_PASSWORD = "<the team password>"349$env:DEV = "1"350python backend/app.py351```352 353## Deployment354 355The Dockerfile builds a Python 3.11 image with Chromium and runs:356 357```358gunicorn --workers 1 --threads 8 --timeout 300 --chdir backend --bind 0.0.0.0:7860 app:app359```360 361`--workers 1` is **required, not a tuning choice**: every job store (uploader,362tagger, PPT, prompt prep) and the CMS credential store live in process memory, so363a second worker would serve requests that can't see the first worker's jobs or364logins. Scale with `--threads`.365 366## Security notes367 368- `state_*.json` and `database/tagger_sessions/*.json` hold live CMS bearer tokens — treat369  them like passwords. They're gitignored and dockerignored, and a background sweep370  deletes any older than 24 hours. In-memory credentials expire after 20 hours and371  job records after 1 hour.372- `SECRET_KEY` must never be a shared constant: this repo is public, and a known373  key lets anyone forge a signed session cookie and walk past the login gate. That374  is why it is generated per machine into `.env` rather than committed.375- The app-gate password is compared with `secrets.compare_digest`, so it can't be376  recovered one character at a time by timing the response.377- Uploads are capped at 25 MB (`MAX_CONTENT_LENGTH`); images at 10 MB.378- `/usage`, `/usage_export` and the Playwright CMS-login capture are restricted to379  requests from localhost — they don't work through the tunnel.380 381## Notes382 383- Every completed run is logged to `usage_log.csv` (viewable at `/usage`, host only)384  *and* mirrored to a shared Supabase dashboard covering all IL tools. Always log385  through `log_usage()` — tools that pushed directly ended up missing from `/usage`,386  and tools that did both were counted twice.387- **`/history`** is the durable job history, and the one that works on the Space.388  `/usage` reads `usage_log.csv`, which is host-only *and* wiped on every Space389  rebuild (the Dockerfile declares no `VOLUME`), so deployed runs had no visible390  history at all. `/history` reads the Supabase `usage_events` table instead —391  the same rows `log_usage()` already writes — via392  [backend/il_job_history.py](backend/il_job_history.py). It is scoped per SME by393  CMS phone; `?scope=all` shows the whole team but only from localhost, the same394  boundary `/usage` draws. `?tools=all` widens it beyond this app to every IL tool395  writing to that table (the MCP server is ~75% of the rows, so it's off by396  default). The reader lives in its own module because `il_shared_usage.py` is a397  write-only client copied verbatim into every IL tool's folder.398- Result files are kept too, in a **private** Supabase Storage bucket399  (`job-artifacts`) via [backend/il_artifacts.py](backend/il_artifacts.py), so the400  Excel/PPTX from a run survives the 1-hour `STORE_TTL`, a restart and a Space401  rebuild. Uploader, Tagger, Quiz PPT and Resolve TOC all store one; a new tool402  opts in by passing `artifact=(name, bytes)` to `log_usage()` — the same single403  call that already records the run. The bucket is private because result files404  carry question content, unlike the diagrams in the public `question-images`405  bucket. `/history/file` checks the key's owner segment against the caller's CMS406  phone *before* minting a 5-minute signed URL — a signed URL bypasses the login407  gate, so handing one out is handing out the file.408- **Question pattern is a placeholder, on purpose.** CMS started requiring a409  question pattern wherever a question is approved. The real vocabulary isn't410  published yet — it is not in `/lookups` under any code, and a sample of 69 live411  questions across the bank turned up none carrying a real value — so every writer412  stamps `not_applicable` until the list is known. Two things to know before413  changing it:414 415  - **Every path that writes or approves a question has to carry it.** The416    uploader builds its payload from scratch ([`build_payload`](backend/app.py));417    the tagger and Solutions tools do fetch → modify → PUT, which faithfully418    preserves the *empty* list most of the bank is in, so they stamp it via419    `push_tags.ensure_question_pattern()`. Approval is the subtle one:420    `PUT /questions/{id}/status` carries only `{"status": ...}`, so a question421    sitting on an empty pattern is rejected on a field that call never sends —422    `approve_tags.approve_one()` fills it with a full PUT *before* running the423    status transitions.424  - **It never overwrites a real value**, only a missing or blank one. The425    placeholder exists to be replaced; a tagging or solutions run must not undo a426    pattern an SME set in CMS. In `push_partial` it is stamped only once a PUT is427    already happening, so a row with no tag columns filled still skips rather than428    writing to questions nobody asked to touch.429 430  When the real values arrive, change `DEFAULT_QUESTION_PATTERN` in431  [backend/ai_tagger/push_tags.py](backend/ai_tagger/push_tags.py) and the literal432  in `app.py`'s `build_payload` together — they must agree, or approvability433  depends on which tool touched the question last. If it turns out to have a434  `/lookups` code, wire it there instead so it stays in sync like `objective` and435  `bloom_taxonomy` do. Note `question_patterns` holds string **slugs**, not the436  UUIDs those other lookup fields store. `tagger.py`'s Playwright automation is437  **not** covered — it drives the CMS UI, so a required dropdown there needs the438  real values.439- Word math conversion uses Microsoft Office's `OMML2MML.XSL`; on a machine without440  Office, formulas fall back to `[FORMULA]` placeholders.441- Optional dependencies degrade gracefully: if `pandas`/`playwright`/`python-pptx`442  are missing, the affected pages say so instead of the app failing to start.443