Datasetpublic

do-me/SemanticFinder

Frontend-only live semantic search with transformers.js App: SemanticFinder GitHub: do-me/SemanticFinder This is the HF data repo for indexed texts, ready-to-import in SemanticFinder. The files contain the original text, text chunks and their embeddings. Catalogue filesize textTitle textAuthor textYear textLanguage URL modelName quantized splitParam splitType characters chunks wordsToAvoidAll wordsToCheckAll wordsToAvoidAny wordsToCheckAny… See the full description on the dataset page: https://huggingface.co/datasets/do-me/SemanticFinder.

sourceHugging Facemitupdated 1y agoView on Hugging Face

14likes922downloads

Dataset Card

<p align="center"> <a href="https://do-me.github.io/SemanticFinder/"> <img src="https://github.com/do-me/SemanticFinder/assets/47481567/4522ab9d-08f4-4f4c-92db-dbf14ccb2b70" width="320" alt="SemanticFinder"> </a> <h1 align="center">Frontend-only live semantic search with transformers.js</h1> </p>

—App: [SemanticFinder](https://do-me.github.io/SemanticFinder/)

—GitHub: [do-me/SemanticFinder](https://github.com/do-me/SemanticFinder)

This is the HF data repo for indexed texts, ready-to-import in SemanticFinder. The files contain the original text, text chunks and their embeddings.

Catalogue

filesize	textTitle	textAuthor	textYear	textLanguage	URL	modelName	quantized	splitParam	splitType	characters	chunks	exportDecimals	lines	textNotes	textSourceURL	filename
11.45	King James Bible		None	en	https://do-me.github.io/SemanticFinder/?hf=KingJamesBible_6434a78d	TaylorAI/gte-tiny	True	200	Chars	4556163	23056	2	80496		https://www.holybooks.com/wp-content/uploads/2010/05/The-Holy-Bible-King-James-Version.pdf	KingJamesBible_6434a78d.json.gz
11.92	Don Quijote	Miguel de Cervantes	1605	es	https://do-me.github.io/SemanticFinder/?hf=DonQuijote14a0b44	Xenova/multilingual-e5-base	True	25	Words	1047150	7186	4	12005		https://parnaseo.uv.es/lemir/revista/revista19/textos/quijote_1.pdf	DonQuijote14a0b44.json.gz
13.52	Iliad	Homer	-750	gr	https://do-me.github.io/SemanticFinder/?hf=Iliad_8de5d1ea	Xenova/multilingual-e5-small	True	20	Words	1597139	11848	5	32659	Including modern interpretation	https://www.stipsi.gr/homer/iliada.pdf	Iliad_8de5d1ea.json.gz
15.61	List of the Most Common English Words	Dolph	2012	en	https://do-me.github.io/SemanticFinder/?hf=ListoftheMostCommonEnglishWords_70320cde	Xenova/multilingual-e5-base	True	\n	Regex	210518	25322	2	25323	GitHub Repo	https://raw.githubusercontent.com/dolph/dictionary/master/popular.txt	ListoftheMostCommonEnglishWords_70320cde.json.gz
2.58	Divina Commedia	Dante	1321	it	https://do-me.github.io/SemanticFinder/?hf=DivinaCommediad5a0fa67	Xenova/multilingual-e5-base	True	50	Words	383782	1179	5	6225		http://www.letteratura-italiana.com/pdf/divina%20commedia/08%20Inferno%20in%20versione%20italiana.pdf	DivinaCommediad5a0fa67.json.gz
4.78	Das Kapital	Karl Marx	1867	de	https://do-me.github.io/SemanticFinder/?hf=DasKapitalc1a84fba	Xenova/multilingual-e5-small	True	80	Words	2003807	3164	5	28673		https://ia601605.us.archive.org/13/items/KarlMarxDasKapitalpdf/KAPITAL1.pdf	DasKapitalc1a84fba.json.gz
1.74	IPCC Report 2023	IPCC	2023	en	https://do-me.github.io/SemanticFinder/?hf=IPCCReport2023_2b260928	Supabase/bge-small-en	True	200	Chars	307811	1566	5	3230	state of knowledge of climate change	https://report.ipcc.ch/ar6syr/pdf/IPCCAR6SYR_LongerReport.pdf	IPCCReport2023_2b260928.json.gz
0.74	Alice’s Adventures in Wonderland	Lewis Carroll	1865	en	https://do-me.github.io/SemanticFinder/?hf=Alice’sAdventuresinWonderland316cc783	Xenova/bge-small-en-v1.5	True	140	Chars	144333	1047	5	1784	Project Gutenberg	https://www.gutenberg.org/files/11/11-h/11-h.htm	Alice’sAdventuresinWonderland316cc783.json.gz
0.46	REGULATION (EU) 2023/138	European Commission	2022	en	https://do-me.github.io/SemanticFinder/?hf=REGULATION(EU)2023138c00e7ff6	Supabase/bge-small-en	True	25	Words	76809	424	5	1323		https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32023R0138&qid=1704492501351	REGULATION(EU)2023138c00e7ff6.json.gz
8.67	List of the Most Common English Words	Dolph	2012	en	https://do-me.github.io/SemanticFinder/?hf=ListoftheMostCommonEnglishWords_0d1e28dc	Xenova/bge-small-en-v1.5	True	\n	Regex	210518	25322	2	25323	GitHub Repo	https://raw.githubusercontent.com/dolph/dictionary/master/popular.txt	ListoftheMostCommonEnglishWords_0d1e28dc.json.gz
0.07	Universal Declaration of Human Rights	United Nations	1948	en	https://do-me.github.io/SemanticFinder/?hf=UniversalDeclarationofHumanRights_0a7da79a	TaylorAI/gte-tiny	True	\nArticle	Regex	8623	63	5	109	30 articles	https://www.un.org/en/about-us/universal-declaration-of-human-rights	UniversalDeclarationofHumanRights_0a7da79a.json.gz
0.06	Hansel and Gretel	Brothers Grimm	1812	en	https://do-me.github.io/SemanticFinder/?hf=HanselandGretel_4de079eb	TaylorAI/gte-tiny	True	100	Chars	5304	55	5	9		https://www.grimmstories.com/en/grimmfairy-tales/hanseland_gretel	HanselandGretel_4de079eb.json.gz
25.52	King James Bible		None	en	https://do-me.github.io/SemanticFinder/?hf=KingJamesBible_7ebed4c7	TaylorAI/gte-tiny	True	\{([^}]+)\}	Regex	4556163	58522	2	80496		https://www.holybooks.com/wp-content/uploads/2010/05/The-Holy-Bible-King-James-Version.pdf	KingJamesBible_7ebed4c7.json.gz
25.56	King James Bible		None	en	https://do-me.github.io/SemanticFinder/?hf=KingJamesBible_24f6dc4c	TaylorAI/gte-tiny	True	200	Chars	4556163	23056	5	80496		https://www.holybooks.com/wp-content/uploads/2010/05/The-Holy-Bible-King-James-Version.pdf	KingJamesBible_24f6dc4c.json.gz
39.32	Les Misérables	Victor Hugo	1862	fr	https://do-me.github.io/SemanticFinder/?hf=LesMisérables2239df51	Xenova/multilingual-e5-base	True	25	Words	3236941	19463	5	74491	All five acts included	https://beq.ebooksgratuits.com/vents/Hugo-miserables-1.pdf	LesMisérables2239df51.json.gz
66.33	Worm	wildbow	2013	en	https://do-me.github.io/SemanticFinder/?hf=Worm_cb8411c1	TaylorAI/gte-tiny	True	100	Chars	9753453	100102	5	237769	Worm, scraped using web2epub, converted to markdown with pandoc.	https://parahumans.wordpress.com	Worm_cb8411c1.json.gz
122.11	A Practical Guide to Evil	ErraticErrata	2022	en	https://do-me.github.io/SemanticFinder/?hf=APracticalGuidetoEvil_fe44ca33	TaylorAI/gte-tiny	True	100	Chars	17940122	183772	5	373823	A Practical Guide to Evil, Turned epub to text with pandoc.	https://practicalguidetoevil.wordpress.com/table-of-contents/	APracticalGuidetoEvil_fe44ca33.json.gz
0.22	196 Countries	Brittanica	2024	en	https://do-me.github.io/SemanticFinder/?hf=196Countriese0118b61	Xenova/jina-embeddings-v2-base-en	True	\n	Regex	1932	197	3	196	Embedding experiment	https://www.britannica.com/topic/list-of-countries-1993160	196Countriese0118b61.json.gz
0.62	Numbers from 0 to 1000		None		https://do-me.github.io/SemanticFinder/?hf=Numbersfrom0to1000_ae7716dc	Xenova/jina-embeddings-v2-base-en	True	,	Regex	4894	1002	2	1	Embedding experiment		Numbersfrom0to1000_ae7716dc.json.gz
100.96	Collection of 100 books	Various Authors	1890	en	https://do-me.github.io/SemanticFinder/?hf=Collectionof100booksdd80b04b	Xenova/bge-small-en-v1.5	True	100	Words	55705582	158957	2	1085035	US Public Domain Books (English)	https://huggingface.co/datasets/storytracer/US-PD-Books/tree/main/data	Collectionof100booksdd80b04b.json.gz
1.2	1910 Unicode Emojis	Unicode, Inc.	2025	en	https://do-me.github.io/SemanticFinder/?hf=1910UnicodeEmojis_9e4e14e7	TaylorAI/gte-tiny	True	\n	Regex	120900	3179	2	1911		https://unicode.org/emoji/charts/emoji-list.html	1910UnicodeEmojis_9e4e14e7.json.gz
0.06	The Elements	Keith Enevoldsen	2025	en	https://do-me.github.io/SemanticFinder/?hf=TheElements70a4ca60	TaylorAI/gte-tiny	True	\n	Regex	14347	133	2	119	Descriptions, Uses and Occurrences	https://elements.wlonk.com/ElementUses.htm	TheElements70a4ca60.json.gz
87.55	New Hampshire Revised Statutes Annotated	State of New Hampshire	2025	en	https://do-me.github.io/SemanticFinder/?hf=NewHampshireRevisedStatutesAnnotated_95bddb2d	TaylorAI/gte-tiny	True	200	Chars	37351243	188798	2	258676	New Hampshire state laws	https://gc.nh.gov/rsa/html/nhtoc.htm	NewHampshireRevisedStatutesAnnotated_95bddb2d.json.gz

Example

Once loaded in SemanticFinder it takes around 2 seconds to search through the whole bible! Try it out.

1.Click on one of the example URLs of your choice.
2.Once the index loaded, simply enter something you want to search for and hit "Find". The results will appear almost instantly.

Create SemanticFinder files

1.Just use SemanticFinder as usual and run at least one search so that the index is created. This might take a while if your input is large. E.g. indexing the bible with 200 chars results in ~23k embeddings and takes 15-30 mins with a quantized gte-tiny model.
2.Add the metadata (so other people can find your index) and export the file. Note that you have the freedom to reduce decimals to reduce file size; usually 2 is more than enough. Less than 2 will also do in most cases but if you need highest accuracy go with 5.
3.Create a PR here if you want to see it added in the official collection! For this, upload the index file and this readme file. For generating the additional markdown table line, create an empty directory on your file system, move the index file and the python script create_meta_data_csv_md.py in there and run the python file with python create_meta_data_csv_md.py. It will create a csv and a markdown file. You can ignore the csv and just copy the table row line from the markdown file and add it to the table in this readme. That's it.

Privacy

—This repo is public and shares documents of public interest or documents in the public domain.
—If you have sensitive documents you can still create the index with SemanticFinder and use it only locally. Either you can load the index from disk each time or you host it in your local network and add the URL in SemanticFinder.

Use cases

Standard use case

Search for most similar words/sentences/paragraphs/pages in any text. Just imagine CTRL+F could find related words and not only the exact same one you used! If you're working on the same text repeatedly you can save the index and reuse it.

Also, there is the option of summarizing the results with generative AI like Qwen models right in your browser or connecting any other LLM with Ollama.

Creative use cases

—Emoji Picker - Find the perfect emojis with semantic search

Advanced use cases

—Translate words with multilingual embeddings or see which words out of a given list are most similar to your input word. Using e.g. the index of ~30k English words you can use more than 100 input languages to query! Note that here the expert settings change so that only the first match is displayed.
—English synonym finder, using again the index of ~30k English words but with slightly better (and smaller) English-only embeddings. Same expert settings here.
—The universal index idea, i.e. use the 30k English words index and do not inference for any new words. In this way you can perform instant semantic search on unknown / unseen / not indexed texts! Use this URL and add then copy and paste any text of your choice into the text field. Inferencing any new words is turned off for speed gains.
—A hybrid version of the universal index where you use the 30k English words as start index but then "fill up" with all the additional words the index doesn't know yet. For this option just use this URL where the inferencing is turned on again. This yields best results and might be a good compromise assuming that new texts generally don't have that many new words. Even if it's a couple of hundreds (like in a particular research paper in a niche domain) inferencing is quite fast.