Botai666/Medical_VLM_Sycophancy

This the official data hosting repository for paper "EchoBench: Benchmarking Sycophancy in Medical Large Vision Language Models".

============open-source_models============

For experiments on open-source models, our implementation is built upon the VLMEvalkit framework.

1. 1.Navigate to the VLMEval directory

1. 1.Set up the environment by running: "pip install -e ."

1. 1.Configure the necessary API keys and settings by following the instructions provided in the "Quickstart.md" file of VLMEvalkit.

1. 1.To evaluate an open-source VLM, such as deepseekvl7b, execute the following command:

AUTOSPLIT=1 torchrun --nproc-per-node=1 run.py --model deepseekvl_7b --data EchoBench --verbose --mode infer

The output will be saved to "deepseekvl7b_EchoBench. xlsx"

1. 1.Navigate to the "evaluation" directory in "apitest" and run: "extractanswer.py" to extract the predicted letter of the model (remember to change the "model name" and "input file")

1. 1.Execute "statistics.py" to get the performance metric of each model

============proprietary_models============

1. 1.Navigate to the api_test directory

1. 1.Run the script: "python localize_dataset.py" （Remember to give the path to EchoBench.tsv)

1. 1.Execute the corresponding Python script for each model. Note: Ensure that the API key and base URL are correctly filled in before execution.

Take GPT-4.1 for example, run "gpt.py", the output will be saved to "gpt-4-1-EchoBench.xlsx", then run "extract_answer.py" to extract the predicted letter of the model.

1. 1.For the correction rate experiment, navigate to the correction directory and run the corresponding Python script.

Take GPT-4.1 for example, run "gptcorrection.py", the output will be saved to "Multiturn-gpt-4-1-EchoBench.xlsx", then run "extractanswer_correction.py" to extract the predicted letter of the model.

1. 1.Execute "statistics.py" and "statistics_correction.py" to get the performance metric of each model