badayvedat/LLaVA

1### ScienceQA2 3#### Prepare Data41. Please see ScienceQA [repo](https://github.com/lupantech/ScienceQA) for setting up the dataset.52. Generate ScienceQA dataset for LLaVA conversation-style format.6 7```Shell8python scripts/convert_sqa_to_llava.py \9    convert_to_llava \10    --base-dir /path/to/ScienceQA/data/scienceqa \11    --prompt-format "QCM-LEA" \12    --split {train,val,minival,test,minitest}13```14 15#### Training16 171. Pretraining18 19You can download our pretrained projector weights from our [Model Zoo](), or train your own projector weights using [`pretrain.sh`](https://github.com/haotian-liu/LLaVA/blob/main/scripts/pretrain.sh).20 212. Finetuning22 23See [`finetune_sqa.sh`](https://github.com/haotian-liu/LLaVA/blob/main/scripts/finetune_sqa.sh).24 25#### Evaluation26 271. Multiple-GPU inference28You may evaluate this with multiple GPUs, and concatenate the generated jsonl files.  Please refer to our script for [batch evaluation](https://github.com/haotian-liu/LLaVA/blob/main/scripts/sqa_eval_batch.sh) and [results gathering](https://github.com/haotian-liu/LLaVA/blob/main/scripts/sqa_eval_gather.sh).29 302. Single-GPU inference31 32(a) Generate LLaVA responses on ScienceQA dataset33 34```Shell35python -m llava.eval.model_vqa_science \36    --model-path liuhaotian/llava-lcs558k-scienceqa-vicuna-13b-v1.3 \37    --question-file /path/to/ScienceQA/data/scienceqa/llava_test_QCM-LEA.json \38    --image-folder /path/to/ScienceQA/data/scienceqa/images/test \39    --answers-file vqa/results/ScienceQA/test_llava-13b.jsonl \40    --conv-mode llava_v141```42 43(b) Evaluate the generated responses44 45```Shell46python eval_science_qa.py \47    --base-dir /path/to/ScienceQA/data/scienceqa \48    --result-file vqa/results/ScienceQA/test_llava-13b.jsonl \49    --output-file vqa/results/ScienceQA/test_llava-13b_output.json \50    --output-result vqa/results/ScienceQA/test_llava-13b_result.json \51```52 53For reference, we attach our prediction file [`test_sqa_llava_lcs_558k_sqa_12e_vicuna_v1_3_13b.json`](https://github.com/haotian-liu/LLaVA/blob/main/llava/eval/table/results/test_sqa_llava_lcs_558k_sqa_12e_vicuna_v1_3_13b.json) and [`test_sqa_llava_13b_v0.json`](https://github.com/haotian-liu/LLaVA/blob/main/llava/eval/table/results/test_sqa_llava_13b_v0.json) for comparison when reproducing our results, as well as for further analysis in detail.54