sahil-12kumar/IL_CMS_Tools

IL CMS Tools

A Flask web app for the Infinity Learn content team. It runs as one process — on the host laptop for LAN/tunnel use, or as a Docker container (Hugging Face Space) — and bundles nine tools that all talk to the IL CMS with the SME's own login.

Screen support: the UI is laid out for laptops and desktops, 1280px wide and up. Phone and tablet layouts are not maintained — pages will render but will not lay out sensibly below 1280px. The layout rules live in frontend/static/il-laptop.css.

The tools

Quick start

pip install -r requirements.txt
playwright install chromium        # needed by Quiz PPT, TOC lookups, make_template.py
python backend/app.py

Then open http://127.0.0.1:7860, or share the printed Network URL with the team. On Windows, start_app_only.bat does the same thing and tags the run for the shared usage dashboard.

First run generates a SECRET_KEY and a random ACCESS_PASSWORD, writes both to a gitignored .env in the repo root, and prints the password in the startup banner. Edit .env to set a password the team can remember. Nothing needs to be configured by hand before the first launch.

Signing in — two separate layers

1. 1.App gate (/login) — the shared ACCESS_PASSWORD. Keeps the LAN/tunnel URL from being open to anyone who finds it. Set ACCESS_PASSWORD= (empty) to run with no gate at all.
2. 2.CMS login — each user enters their own CMS phone + password on the Home page. That single login fills the server-side credential store (Uploader, PPT, Resolve TOC, Diagram Upload) and writes the per-browser session file the Tagger / AI Tagging / LO Classifier use. Sessions last 20 hours.

The CMS token is held server-side, never in the browser cookie — the cookie only carries a client id. Restarting the server therefore disconnects everyone, and they reconnect on the Home page.

Verify and Solutions can additionally connect their own CMS account, independent of the Home login, so a reviewer can push under a different identity without disturbing the other tools. Each has its own Connect/Disconnect control.

1. Uploader (/uploader)

1. 1.Upload a file — .xlsx, .docx, .md, or .zip (markdown + images) — or paste question text straight into the page. Sample templates are downloadable on the page; the formats are documented in samples/README.md.
2. 2.Review — parsing runs in the background with live progress; each question gets a confidence flag and a list of issues (no answer key, fewer than four options, …). An answer key can be pasted separately and merged by question number.
3. 3.Upload to CMS — questions are pushed one-by-one as drafts; a results report with the new question IDs downloads as Excel.

Math (OMML, MathType OLE, LaTeX) is converted to MathML; images become embedded base64 or Supabase URLs. Numerical questions carry their answer range through validation into the payload.

upload_md.py is the same engine as a CLI — --check, --key/--inject, --upload — for driving a question bank from the terminal.

2. Tagger (/tagger)

The run engine is the fast QB REST API tagger (api_tagger.py): per question it resolves chapter/topic/subtopic names to the chosen exam's node IDs, PUTs the tags + metadata, then drives the status workflow to approved or under review. It runs sequentially — the worker-count control is legacy and no longer parallelises. tagger.py is kept for its Excel/session helpers and the CMS login, plus the older Playwright engine.

Tagging Excel columns (any order; only question_id is mandatory):

question_id | subject | chapter | topic | subtopic | grade
objective | difficulty | bloom_taxonomy | relevance
minitopic | micro_topic | concept_level | source | syllabus
author | tags | exam | year | month | date | toc

Tracking columns (Status, Run Details, Attempts, Last Worker, Last Updated) are added automatically and updated row-by-row, so the downloaded "updated Excel" doubles as the run report. Rows marked processed are skipped on re-runs — after a run with failures, just run again with the Question IDs box empty to retry only the failed rows.

Steps on the page: upload the tagging Excel → pick the exam TOC pill (NEET, JEE Main, …) and approval mode → Run. Live log + progress; on finish you get a failed-questions report and the updated Excel. Each user gets their own run; a second run for the same user is refused while one is in progress.

Approval mode is admin-only: Full Approve (the workflow all the way to approved) is restricted to the admin CMS phone 9939921144; everyone else is locked to Under Review and the radio is hidden. The check is enforced in /tagger/start server-side, not just in the UI. can_approve gates the radio in tagger.html / tag.html.

python backend/make_template.py regenerates database/tagging_template.xlsx by scraping the live CMS form with Playwright, so every dropdown column carries valid in-Excel options. /tagger/refresh_template does the same from the page.

3. The AI tools — /tag, /verify, /solutions, /lo

These four share one design and need no API key and no paid AI account:

1. 1.The app fetches each question from the CMS and renders it readably — cms_text.py turns MathML into $...$ LaTeX and marks images as [figure], so the AI never sees a half-broken question.
2. 2.It builds one copy-paste prompt that also carries the valid vocabulary (exam TOC, dropdown options, or the LO taxonomy).
3. 3.The SME pastes it into claude.ai / ChatGPT on their own subscription and pastes the JSON reply back.
4. 4.The app hard-validates every value against the real taxonomy before anything is written — a row that doesn't resolve is rejected, not guessed.

/verify and /solutions are solve-blind: the stored answer is withheld from the prompt and kept server-side, so a disagreement is real signal rather than the AI agreeing with what it was shown. Key writes are type-aware — MCQs store 1-based option indices, numeric types store the literal value — and are re-encoded from the live record at push time.

4. Quiz PPT (/ppt)

Paste QIDs, optionally upload your own .pptx template, and each question + options are rendered locally with MathJax in one persistent headless page and dropped onto the branded template slide. Runs as a background job with a live log. ppt_api.py also works as a standalone CLI.

Chapter / Topic title slides. Instead of a flat QID list you can paste rows straight from a spreadsheet — Chapter Topic QID Year/Exam (the chapter and topic index numbers are fine to include). Each new chapter emits a chapter title slide (with a NEET PYQ tag), each new topic a topic title slide, then the question slides beneath them. On each question slide the year sits in a top-left box and chapter + topic in a bottom-right box. A plain flat QID list (one per line) still works and produces question slides only. ppt_api.parse_tabular groups the rows by chapter then topic in first-seen order.

Chapter-wise output (ZIP). When a pasted table produces multiple chapters, the generator instead builds one branded `.pptx` per chapter and delivers them as a single ZIP download. To put one question into several chapters, just repeat its row under each chapter — it is fetched/rendered once and reused in every chapter deck it appears in. (ppt_api.generate_chapter_wise does the fan-out; the flat plain-QID mode stays a single file.)

5. Diagram Upload (/upload_image)

Uploads a PNG/JPG/GIF/WebP (≤10 MB) to the Supabase question-images bucket and returns its permanent public URL. Requires `SUPABASE_URL` + `SUPABASE_KEY` — without them every attempt returns "Image storage not configured". This is the one tool that doesn't need a CMS connection.

6. Resolve TOC (/resolve_toc)

Paste up to 500 question IDs and get an Excel of their live chapter / topic / subtopic / grade names. It probes the community API separately from the QB API, because the community API invalidates tokens hours earlier — otherwise the page would say "Connected" and silently resolve every name to blank.

Exam TOC / TOC Chain / Chapter / Topic all come from row 1 of the question's "Visible to students in this exams" list — the numbered list CMS shows on the question page, e.g. for ILQ-1836594:

1. CBSE Grade 7 > Mathematics > Exponents and Powers > Laws of exponents
2. CBSE Grade 8 > Mathematics > Power Play. > Laws of Exponents
3. Grade 8-AP State Board > Mathematics > Exponents and Powers(Inactive) > ...

Rows read exam > subject > chapter > topic, and the TOC Chain column reproduces row 1 verbatim. Those rows come from the same get_navigation_details_for_subtopic call the names already needed — no extra request. Exam TOCs include board/grade TOCs, not only JEE/NEET.

The order is the API's own, which is the order CMS numbers them in. It is deliberately not re-sorted by the question's stored chapter_id: that ID often points at a later row, which is what made earlier versions report a chapter the SME wasn't expecting. (Inactive) in row 3 above is part of the chapter name, not a status flag — every row in this panel is student-visible, so no row is filtered out. Subject comes from the row too, overriding the master name (Mathematics, not Maths). The subtopic keeps its own column and is absent from the chain, since CMS's list stops at the topic.

Questions with no rows at all fall back to the master TOC names and a blank Exam TOC.

Speed. A resolve is two API calls per question and nothing else, so it is pure I/O wait. Three things keep a 500-ID batch quick: RESOLVE_TOC_WORKERS threads (default 12, clamped 1–32 — raise it if runs stay clean, drop it if rows start coming back as ERROR), one pooled requests.Session so calls reuse connections instead of paying a TLS handshake each, and a per-job cache of nav responses keyed by subtopic. That last one matters most in practice: a pasted batch is usually a single chapter, so a few subtopics repeat across every row. The cache is per job, never process-wide, so a TOC edited in CMS shows up on the next run.

Repo layout

IL_CMS_Tools/
├── backend/                  # All Python code
│   ├── app.py                #   Flask app — every route for all nine tools
│   ├── tagger.py             #   CMS login, tagging-Excel helpers, legacy Playwright engine
│   ├── api_tagger.py         #   Fast QB REST tagging engine (the one /tagger runs)
│   ├── ai_tagger/            #   Shared CMS API primitives
│   │   ├── push_tags.py      #     QB headers from a state file, fetch, resolve_ids, push_one
│   │   ├── approve_tags.py   #     status workflow -> under review / approved
│   │   └── extract_toc.py    #     exam TOC name -> node id lookups (shared service account)
│   ├── ai_assist.py          #   AI Tagging bridge — prompt build + JSON validation -> Excel
│   ├── tag_assist.py         #   Curated exam TOC + merged CMS option lists for /tag
│   ├── verify_assist.py      #   Solve-blind answer verification + adjudicator pass
│   ├── solution_assist.py    #   Solution authoring + type-aware answer-key writeback
│   ├── lo_assist.py          #   LO Classifier — taxonomy validation + per-subject bank
│   ├── cms_text.py           #   CMS question HTML/MathML -> readable text for prompts
│   ├── ppt_api.py            #   Quiz PPT generator (MathJax render -> branded slides)
│   ├── make_template.py      #   Scrapes the CMS form -> database/tagging_template.xlsx
│   ├── upload_md.py          #   CLI wrapper around the uploader engine
│   ├── mathtype_mtef.py      #   MathType OLE equation streams -> MathML
│   ├── il_shared_usage.py    #   Fire-and-forget mirror of usage events to Supabase
│   ├── build_exam_toc.py     #   SME TOC sheets  -> database/exam_toc.json (+ review workbook)
│   ├── build_lo_taxonomy.py  #   SME LO sheets   -> database/lo_taxonomy.json
│   ├── build_cms_options.py  #   CMS /lookups    -> database/cms_options.json snapshot
│   └── vendor/               #   OMML2MML.XSL — Word-equation conversion (committed)
├── frontend/
│   ├── templates/            # home, uploader, tagger, tag, verify, solutions, lo,
│   │                         # ppt, upload_image, resolve_toc, usage, login, _nav
│   └── static/               # il-laptop.css (layout system), il-common.js
├── database/                 # Data the tools read/write
│   ├── exam_toc.json         #   Curated per-exam TOC, snapped to exact CMS names
│   ├── lo_taxonomy.json      #   Subject -> chapter -> topic -> subtopic -> LO
│   ├── cms_options.json      #   Fallback metadata option lists
│   ├── ppt_template.pptx     #   Default Quiz PPT template
│   ├── tagging_template.xlsx #   Generated tagging template (gitignored)
│   └── ...                   #   runtime state (gitignored) — see below
├── samples/                  # Downloadable sample question + tagging files
├── tests/                    # pytest suite (imports the app from backend/)
├── Dockerfile                # HF Space / container image
└── requirements.txt

Runtime-only, all gitignored: .env (repo root), state_*.json, and under database/: tagger_sessions/, tagger_reports/, tagger_screenshots/, lo_bank/, usage_log.csv.

Rebuilding the data files

Run on the host laptop with a fresh CMS login, then commit the JSON:

python backend/build_exam_toc.py       # when the curated TOC sheets or the CMS tree change
python backend/build_lo_taxonomy.py    # when an LO sheet changes
python backend/build_cms_options.py    # when CMS changes the metadata dropdowns
python backend/make_template.py        # when the CMS question form changes

build_exam_toc.py also writes database/exam_toc_review.xlsx — filter status != exact, fix genuine mis-snaps in exam_toc_aliases.xlsx, and re-run.

Configuration (environment variables)

Every value below can live in .env in the repo root (auto-created, gitignored) or in the real environment — a real environment variable always wins. Docker / HF Space deployments supply them as Space secrets.

Dev vs production endpoints

Every CMS host name lives in backend/il_endpoints.py and defaults to the production stack — with nothing set, every tool reads and writes real CMS content. Nothing else in the code hardcodes a host, so pointing the whole app at dev/preprod is four lines in .env:

IL_QB_API=https://newqbapi.devinfinitylearn.in
IL_GATEWAY_API=https://gatewayapi.devinfinitylearn.in
IL_COMMUNITY_API=https://cmspreprodapi.devinfinitylearn.in
IL_COMMUNITY_WEB=https://cmspreprod.devinfinitylearn.in

IL_COMMUNITY_WEB is a different host from IL_COMMUNITY_API, not a duplicate: the API answers on communityapi.infinitylearn.com, while the CMS a browser loads is community.infinitylearn.com (no api). The web value is what Playwright opens for the Tagger and make_template.py, and what fills referer headers — pointing it at the API host breaks both tools.

# Overriding for one run (PowerShell)
$env:ACCESS_PASSWORD = "<the team password>"
$env:DEV = "1"
python backend/app.py

Deployment

The Dockerfile builds a Python 3.11 image with Chromium and runs:

gunicorn --workers 1 --threads 8 --timeout 300 --chdir backend --bind 0.0.0.0:7860 app:app

--workers 1 is required, not a tuning choice: every job store (uploader, tagger, PPT, prompt prep) and the CMS credential store live in process memory, so a second worker would serve requests that can't see the first worker's jobs or logins. Scale with --threads.

Security notes

• —state_*.json and database/tagger_sessions/*.json hold live CMS bearer tokens — treat them like passwords. They're gitignored and dockerignored, and a background sweep deletes any older than 24 hours. In-memory credentials expire after 20 hours and job records after 1 hour.
• —SECRET_KEY must never be a shared constant: this repo is public, and a known key lets anyone forge a signed session cookie and walk past the login gate. That is why it is generated per machine into .env rather than committed.
• —The app-gate password is compared with secrets.compare_digest, so it can't be recovered one character at a time by timing the response.
• —Uploads are capped at 25 MB (MAX_CONTENT_LENGTH); images at 10 MB.
• —/usage, /usage_export and the Playwright CMS-login capture are restricted to requests from localhost — they don't work through the tunnel.

Notes

• —Every completed run is logged to usage_log.csv (viewable at /usage, host only) and mirrored to a shared Supabase dashboard covering all IL tools. Always log through log_usage() — tools that pushed directly ended up missing from /usage, and tools that did both were counted twice.
• —`/history` is the durable job history, and the one that works on the Space. /usage reads usage_log.csv, which is host-only and wiped on every Space rebuild (the Dockerfile declares no VOLUME), so deployed runs had no visible history at all. /history reads the Supabase usage_events table instead — the same rows log_usage() already writes — via backend/il_job_history.py. It is scoped per SME by CMS phone; ?scope=all shows the whole team but only from localhost, the same boundary /usage draws. ?tools=all widens it beyond this app to every IL tool writing to that table (the MCP server is ~75% of the rows, so it's off by default). The reader lives in its own module because il_shared_usage.py is a write-only client copied verbatim into every IL tool's folder.
• —Result files are kept too, in a private Supabase Storage bucket (job-artifacts) via backend/il_artifacts.py, so the Excel/PPTX from a run survives the 1-hour STORE_TTL, a restart and a Space rebuild. Uploader, Tagger, Quiz PPT and Resolve TOC all store one; a new tool opts in by passing artifact=(name, bytes) to log_usage() — the same single call that already records the run. The bucket is private because result files carry question content, unlike the diagrams in the public question-images bucket. /history/file checks the key's owner segment against the caller's CMS phone before minting a 5-minute signed URL — a signed URL bypasses the login gate, so handing one out is handing out the file.
• —Question pattern is a placeholder, on purpose. CMS started requiring a question pattern wherever a question is approved. The real vocabulary isn't published yet — it is not in /lookups under any code, and a sample of 69 live questions across the bank turned up none carrying a real value — so every writer stamps not_applicable until the list is known. Two things to know before changing it:

• —Every path that writes or approves a question has to carry it. The uploader builds its payload from scratch (`build_payload`); the tagger and Solutions tools do fetch → modify → PUT, which faithfully preserves the empty list most of the bank is in, so they stamp it via push_tags.ensure_question_pattern(). Approval is the subtle one: PUT /questions/{id}/status carries only {"status": ...}, so a question sitting on an empty pattern is rejected on a field that call never sends — approve_tags.approve_one() fills it with a full PUT before running the status transitions.
• —It never overwrites a real value, only a missing or blank one. The placeholder exists to be replaced; a tagging or solutions run must not undo a pattern an SME set in CMS. In push_partial it is stamped only once a PUT is already happening, so a row with no tag columns filled still skips rather than writing to questions nobody asked to touch.

When the real values arrive, change DEFAULT_QUESTION_PATTERN in backend/ai_tagger/push_tags.py and the literal in app.py's build_payload together — they must agree, or approvability depends on which tool touched the question last. If it turns out to have a /lookups code, wire it there instead so it stays in sync like objective and bloom_taxonomy do. Note question_patterns holds string slugs, not the UUIDs those other lookup fields store. tagger.py's Playwright automation is not covered — it drives the CMS UI, so a required dropdown there needs the real values.

• —Word math conversion uses Microsoft Office's OMML2MML.XSL; on a machine without Office, formulas fall back to [FORMULA] placeholders.
• —Optional dependencies degrade gracefully: if pandas/playwright/python-pptx are missing, the affected pages say so instead of the app failing to start.