nmac/lex_fridman_podcast_semantic_search

lex-semantic-search

Gradio application for performing semantic search on Lex Fridman podcast transcripts.

Dataset

The Gradio application is pre-loaded with chunks (chunk size is 25 contiguous entries) and embeddings for dataset nmac/lex_fridman_podcast.

Usage

1. 1.Set up virtual environment with the required dependencies:

python -m venv lex-semantic-search
source lex-semantic-search/bin/activate 
pip install -r requirements.txt     # for GPU
pip install -r requirements_cpu.txt # for CPU

1. 1.Run the application locally using the following command:

python app.py

1. 1.Access the application by opening your web browser and navigating to http://localhost:7860.

1. 1.In the application interface, adjust the input settings according to your needs:
2. 2.Query: Enter a query to search for relevant podcast transcript chunks related to it.
3. 3.Chunk Size: Adjust the chunk size. (Fixed to 25)
4. 4.Embeddings Generator: Select the embeddings generator to use. (Fixed to `sentence-transformers/multi-qa-mpnet-base-dot-v1`)
5. 5.Retriever Method: Select the retriever method. (Fixed to `FAISS`)
6. 6.Number of Chunks to Retrieve: Set the number of chunks to retrieve.

1. 1.Click the "Submit" button to retrieve the chunks that match your settings and query. The results will be displayed in a table.