internlm/MindSearch

1---2title: MindSearch3emoji: 📊4colorFrom: purple5colorTo: yellow6sdk: gradio7sdk_version: 5.7.18app_file: app.py9pinned: false10---11 12<div id="top"></div>13 14<div align="center">15 16<img src="assets/logo.svg" style="width: 50%; height: auto;">17 18[📃 Paper](https://arxiv.org/abs/2407.20183) | [💻 Demo](https://internlm-chat.intern-ai.org.cn/)19 20English | [简体中文](README_zh-CN.md)21 22<https://github.com/user-attachments/assets/44ffe4b9-be26-4b93-a77b-02fed16e33fe>23 24</div>25</p>26 27## ✨ MindSearch: Mimicking Human Minds Elicits Deep AI Searcher28 29## 📅 Changelog30 31- 2024/11/05: 🥳 MindSearch is now deployed on Puyu! 👉 [Try it](https://internlm-chat.intern-ai.org.cn/) 👈32  -  Refactored the agent module based on [Lagent v0.5](https://github.com/InternLM/lagent) for better performance in concurrency.33  -  Improved the UI to embody the simultaneous multi-query search.34 35 36## ⚽️ Build Your Own MindSearch37 38### Step1: Dependencies Installation39 40```bash41git clone https://github.com/InternLM/MindSearch42cd MindSearch43pip install -r requirements.txt44```45 46### Step2: Setup Environment Variables47 48Before setting up the API, you need to configure environment variables. Rename the `.env.example` file to `.env` and fill in the required values.49 50```bash51mv .env.example .env52# Open .env and add your keys and model configurations53```54 55### Step3: Setup MindSearch API56 57Setup FastAPI Server.58 59```bash60python -m mindsearch.app --lang en --model_format internlm_silicon --search_engine DuckDuckGoSearch --asy 61```62 63- `--lang`: language of the model, `en` for English and `cn` for Chinese.64- `--model_format`: format of the model.65  - `internlm_server` for InternLM2.5-7b-chat with local server. (InternLM2.5-7b-chat has been better optimized for Chinese.)66  - `gpt4` for GPT4.67    if you want to use other models, please modify [models](./mindsearch/agent/models.py)68- `--search_engine`: Search engine.69  - `DuckDuckGoSearch` for search engine for DuckDuckGo.70  - `BingSearch` for Bing search engine.71  - `BraveSearch` for Brave search web api engine.72  - `GoogleSearch` for Google Serper web search api engine.73  - `TencentSearch` for Tencent search api engine.74  75  Please set your Web Search engine API key as the `WEB_SEARCH_API_KEY` environment variable unless you are using `DuckDuckGo`, or `TencentSearch` that requires secret id as `TENCENT_SEARCH_SECRET_ID` and secret key as `TENCENT_SEARCH_SECRET_KEY`.76- `--asy`: deploy asynchronous agents.77 78### Step4: Setup MindSearch Frontend79 80Providing following frontend interfaces,81 82- React83 84First configurate the backend URL for Vite proxy.85 86```bash87HOST="127.0.0.1"  # modify as you need88PORT=800289sed -i -r "s/target:\s*\"\"/target: \"${HOST}:${PORT}\"/" frontend/React/vite.config.ts90```91 92```bash93# Install Node.js and npm94# for Ubuntu95sudo apt install nodejs npm96 97# for windows98# download from https://nodejs.org/zh-cn/download/prebuilt-installer99 100# Install dependencies101 102cd frontend/React103npm install104npm start105```106 107Details can be found in [React](./frontend/React/README.md)108 109- Gradio110 111```bash112python frontend/mindsearch_gradio.py113```114 115- Streamlit116 117```bash118streamlit run frontend/mindsearch_streamlit.py119```120 121## 🌐 Change Web Search API122 123To use a different type of web search API, modify the `searcher_type` attribute in the `searcher_cfg` located in `mindsearch/agent/__init__.py`. Currently supported web search APIs include:124 125- `GoogleSearch`126- `DuckDuckGoSearch`127- `BraveSearch`128- `BingSearch`129- `TencentSearch`130 131For example, to change to the Brave Search API, you would configure it as follows:132 133```python134BingBrowser(135    searcher_type='BraveSearch',136    topk=2,137    api_key=os.environ.get('BRAVE_API_KEY', 'YOUR BRAVE API')138)139```140 141## 🐞 Using the Backend Without Frontend142 143For users who prefer to interact with the backend directly, use the `backend_example.py` script. This script demonstrates how to send a query to the backend and process the response.144 145```bash146python backend_example.py147```148 149Make sure you have set up the environment variables and the backend is running before executing the script.150 151## 🐞 Debug Locally152 153```bash154python -m mindsearch.terminal155```156 157## 📝 License158 159This project is released under the [Apache 2.0 license](LICENSE).160 161## Citation162 163If you find this project useful in your research, please consider cite:164 165```166@article{chen2024mindsearch,167  title={MindSearch: Mimicking Human Minds Elicits Deep AI Searcher},168  author={Chen, Zehui and Liu, Kuikun and Wang, Qiuchen and Liu, Jiangning and Zhang, Wenwei and Chen, Kai and Zhao, Feng},169  journal={arXiv preprint arXiv:2407.20183},170  year={2024}171}172```173 174## Our Projects175 176Explore our additional research on large language models, focusing on LLM agents.177 178- [Lagent](https://github.com/InternLM/lagent): A lightweight framework for building LLM-based agents179- [AgentFLAN](https://github.com/InternLM/Agent-FLAN): An innovative approach for constructing and training with high-quality agent datasets (ACL 2024 Findings)180- [T-Eval](https://github.com/open-compass/T-Eval): A Fine-grained tool utilization evaluation benchmark (ACL 2024)181