JamesSeegel/lab05-semantic-search
Speak the Patient's Language: semantic search vs keyword search Completed by James Seegel for MIS 752 at UNLV. The teaching corpus and evaluation queries were provided in Dr. Richard Young's lab notebook. My takeaways Sentences within the same topic averaged 0.235 cosine similarity, compared with 0.129 across topics, showing some separation between subjects. Keyword search handled “my refill is not ready at the pharmacy” because “refill,” “ready,” and “pharmacy”… See the full description on the dataset page: https://huggingface.co/datasets/JamesSeegel/lab05-semantic-search.
Speak the Patient's Language: semantic search vs keyword search
Completed by James Seegel for MIS 752 at UNLV.
The teaching corpus and evaluation queries were provided in Dr. Richard Young's lab notebook.
My takeaways
Sentences within the same topic averaged 0.235 cosine similarity, compared with 0.129 across topics, showing some separation between subjects. Keyword search handled “my refill is not ready at the pharmacy” because “refill,” “ready,” and “pharmacy” appear in the matching sentence. For queries without shared vocabulary, keyword MRR was 0.000 versus 0.800 for semantic search. An empty search page could leave patients without useful information even when the right material exists. Overall, semantic MRR was 0.917, compared with 0.500 for keyword search. However, semantic search missed “the reading on my meter jumped since supper,” returning car5, dia5, and car2 instead of the relevant diabetes sentences. I think “meter” was ambiguous, and the model did not connect the meal reference with blood sugar. In an IT help desk, semantic search could connect “it keeps kicking me out” with “session timeout.” I would test similar phrases requiring different solutions before relying on the results.
Method and limitations
The 20 teaching sentences and 12 evaluation queries were provided in Dr. Richard Young's MIS 752 lab notebook. I used sentence-transformers/all-MiniLM-L6-v2 to create normalized embeddings, then ranked results by cosine similarity. The keyword baseline counts shared content words after removing stopwords.
Precision uses relevant results divided by the number actually returned; recall uses relevant results divided by all relevant items in the answer key. MRR uses the first relevant result within the top 3, with zero when none is found.
These results describe this small teaching corpus and its answer key. BM25 and reranking were not evaluated. The examples are course materials and do not validate a clinical search system.
The data files contain the corpus, embeddings, cosine matrix, per-query evaluation, metric summaries, and example results.