WangKaiLin/PipeOwl-1.10.2-tw-wiki-rag

PipeOwl-1.10.2-tw-wiki-rag

[image]

A transformer-free semantic retrieval engine for multilingual wiki retrieval and passage expansion.

It combines:

• —a static embedding field
• —entity normalization
• —phrase-aware tokenization
• —merge-based title retrieval
• —local passage spread from matched wiki entries

Performance Mode Switch

If you need a faster query version, replace the files in the `fast_query` folder with those in the root directory.

The initial loading time will be longer (approximately +4–5 seconds), but the query time can be reduced from around 25 seconds to under 1 second.

如果你需要query更快的版本 把fast_query資料夾內的檔案覆蓋到主目錄下
第一次載入會比較久 載入時間會多個4~5秒 但查詢時間可以從25秒壓到1秒內

Current Limitations

• —Query latency is still dominated by Python-side retrieval logic.
• —entity_alias.json is incomplete and still growing.
• —phrase_lexicon.txt is heuristic and hand-curated.
• —Passage spread is a lightweight local retrieval step, not a full global chunk index.
• —Wiki freshness depends on the included zhwiki dump.

Architecture

• —Static embedding table (V × D)
• —Aligned vocabulary index
• —Linear scoring
• —Pluggable decoder stage

Model Specs

Quickstart

Tested on Python 3.10+

Unzip pipeowl1.10.2.zip

git clone https://huggingface.co/WangKaiLin/PipeOwl-1.10.2-tw-wiki-rag/
cd PipeOwl-1.10.2-tw-wiki-rag/

pip install numpy safetensors

python quickstart_wiki_merge.py

Example:

請見專案內 example.md

範例: Input: bang dream

Output:

• —normalized query: BanG_Dream!
• —top title: BanG_Dream!
• —knowledge spread: a matched passage from the wiki article

See example.md for full examples.

Repository Structure

PipeOwl-1.10.2-tw-wiki-rag/
├─ quickstart_wiki_merge.py #入口 
├─ wiki_retriever_merge.py #wiki資料搜尋
├─ entity_layer.py #同義層邏輯
├─ tokenizer_priority.py #切詞器邏輯
├─ engine.py #pipeowl核心模組入口
├─ entity_alias.json #同義詞資料集 (未完備)
├─ phrase_lexicon.txt #完整詞保護
├─ tokenizer.json #核心tokenizer
├─ pipeowl.safetensors #核心向量矩陣儲存
├─ zhwiki_clean.jsonl #wikidata資料集
└─ wiki_index_merge/ 
   ├─ wiki_merge_meta.json 
   ├─ wiki_titles.txt 
   ├─ wiki_title_tokens.jsonl
   ├─ wiki_title_tokens_offsets.json
   ├─ wiki_token_to_title_ids.json
   └─ wiki_title_offsets.json

License

Code and system components in this repository are released under the MIT License.

Included Wikipedia-derived retrieval data is subject to CC BY-SA 4.0. Please ensure attribution and share-alike compliance if redistributing derived wiki data.