AnTo2209/MarineEVT
MarineEVT Dataset MarineEVT: Advancing Event-Centric Marine Video Understanding via Visual Tool Reasoning 📖 Description MarineEVT is a comprehensive event-centric dataset and benchmark for marine video understanding. It comprises 20,000 richly annotated underwater video question-answer pairs spanning 20 fine-grained dimensions, designed to support semantic, contextualized, spatial-temporal, and causal reasoning in marine environments. The dataset addresses the… See the full description on the dataset page: https://huggingface.co/datasets/AnTo2209/MarineEVT.
01.5k
1{2 "data": [3 {4 "question": "Localize the species with the traits of long slender body with black and white stripes, forked tail in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",5 "answer": "{\"boxes\": [[[\"6.075611114501953\", \"506.46612548828125\", \"725.2152709960938\", \"897.7997436523438\"]], [[\"6.680545806884766\", \"496.44952392578125\", \"724.8888549804688\", \"906.0802612304688\"]], [[\"74.2911605834961\", \"481.9813537597656\", \"725.249755859375\", \"881.7330932617188\"]], [[\"92.12799835205078\", \"478.6889343261719\", \"728.5828247070312\", \"871.3319091796875\"]], [[\"127.72293090820312\", \"472.79052734375\", \"726.3589477539062\", \"872.4901123046875\"]], [[\"132.5176544189453\", \"467.7110290527344\", \"728.1343383789062\", \"870.9744262695312\"]], [[\"128.05780029296875\", \"452.19671630859375\", \"726.2186279296875\", \"867.9617919921875\"]]]}",6 "explanation": {7 "0": [8 "6.075611114501953",9 "506.46612548828125",10 "725.2152709960938",11 "897.7997436523438"12 ],13 "1": [14 "6.680545806884766",15 "496.44952392578125",16 "724.8888549804688",17 "906.0802612304688"18 ],19 "2": [20 "74.2911605834961",21 "481.9813537597656",22 "725.249755859375",23 "881.7330932617188"24 ],25 "3": [26 "92.12799835205078",27 "478.6889343261719",28 "728.5828247070312",29 "871.3319091796875"30 ],31 "4": [32 "127.72293090820312",33 "472.79052734375",34 "726.3589477539062",35 "872.4901123046875"36 ],37 "5": [38 "132.5176544189453",39 "467.7110290527344",40 "728.1343383789062",41 "870.9744262695312"42 ],43 "6": [44 "128.05780029296875",45 "452.19671630859375",46 "726.2186279296875",47 "867.9617919921875"48 ]49 },50 "dimension": "SpatialReasoning",51 "subdimension": "VisualGrounding",52 "question_task": "VideoQuestionAnswering",53 "question_format": "ExactMatchQuestion",54 "video_id": "t05mYAVwaDZ5",55 "video_url": "https://www.instagram.com/tomparkfilms/reel/C9itHDqOnuU",56 "video_scene": "000002_000008",57 "id": 0,58 "original_video_id": "C9itHDqOnuU",59 "turns": [60 {61 "turn_id": 1,62 "id": 0,63 "assistant_response": "[{\"function\": {\"name\": \"object_grounding\", \"arguments\": {\"prompt\": \"A region of a species with long slender body and black and white stripes, forked tail\", \"ground_type\": \"anything\"}}}]",64 "user_query": "Localize the species with the traits of long slender body with black and white stripes, forked tail in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",65 "visual_input": [66 "frames/frame_000000.jpg",67 "frames/frame_000001.jpg",68 "frames/frame_000002.jpg",69 "frames/frame_000003.jpg",70 "frames/frame_000004.jpg",71 "frames/frame_000005.jpg",72 "frames/frame_000006.jpg"73 ],74 "tool_output": "{\"boxes\": [[[\"6.075611114501953\", \"506.46612548828125\", \"725.2152709960938\", \"897.7997436523438\"]], [[\"6.680545806884766\", \"496.44952392578125\", \"724.8888549804688\", \"906.0802612304688\"]], [[\"74.2911605834961\", \"481.9813537597656\", \"725.249755859375\", \"881.7330932617188\"]], [[\"92.12799835205078\", \"478.6889343261719\", \"728.5828247070312\", \"871.3319091796875\"]], [[\"127.72293090820312\", \"472.79052734375\", \"726.3589477539062\", \"872.4901123046875\"]], [[\"132.5176544189453\", \"467.7110290527344\", \"728.1343383789062\", \"870.9744262695312\"]], [[\"128.05780029296875\", \"452.19671630859375\", \"726.2186279296875\", \"867.9617919921875\"]]]}"75 },76 {77 "id": 0,78 "turn_id": 2,79 "user_query": "Localize the species with the traits of long slender body with black and white stripes, forked tail in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",80 "assistant_response": "<answer>{\"boxes\": [[[\"6.075611114501953\", \"506.46612548828125\", \"725.2152709960938\", \"897.7997436523438\"]], [[\"6.680545806884766\", \"496.44952392578125\", \"724.8888549804688\", \"906.0802612304688\"]], [[\"74.2911605834961\", \"481.9813537597656\", \"725.249755859375\", \"881.7330932617188\"]], [[\"92.12799835205078\", \"478.6889343261719\", \"728.5828247070312\", \"871.3319091796875\"]], [[\"127.72293090820312\", \"472.79052734375\", \"726.3589477539062\", \"872.4901123046875\"]], [[\"132.5176544189453\", \"467.7110290527344\", \"728.1343383789062\", \"870.9744262695312\"]], [[\"128.05780029296875\", \"452.19671630859375\", \"726.2186279296875\", \"867.9617919921875\"]]]}</answer>",81 "visual_input": [82 "frames/frame_000000.jpg",83 "frames/frame_000001.jpg",84 "frames/frame_000002.jpg",85 "frames/frame_000003.jpg",86 "frames/frame_000004.jpg",87 "frames/frame_000005.jpg",88 "frames/frame_000006.jpg"89 ]90 }91 ]92 },93 {94 "question": "Localize the species with the traits of long, slender body, dark mottled skin, partially hidden in crevices in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",95 "answer": "{\"boxes\": [[[\"276.62890625\", \"462.6394348144531\", \"594.9517822265625\", \"703.8656005859375\"]], [[\"222.19586181640625\", \"453.16802978515625\", \"638.389892578125\", \"735.9738159179688\"]], [[\"173.78103637695312\", \"424.33538818359375\", \"666.8064575195312\", \"767.375\"]], [[\"120.7812271118164\", \"418.6991882324219\", \"700.930419921875\", \"816.4493408203125\"]], [[\"74.67980194091797\", \"412.57354736328125\", \"721.7605590820312\", \"917.9664306640625\"]]]}",96 "explanation": {97 "0": [98 "276.62890625",99 "462.6394348144531",100 "594.9517822265625",101 "703.8656005859375"102 ],103 "1": [104 "222.19586181640625",105 "453.16802978515625",106 "638.389892578125",107 "735.9738159179688"108 ],109 "2": [110 "173.78103637695312",111 "424.33538818359375",112 "666.8064575195312",113 "767.375"114 ],115 "3": [116 "120.7812271118164",117 "418.6991882324219",118 "700.930419921875",119 "816.4493408203125"120 ],121 "4": [122 "74.67980194091797",123 "412.57354736328125",124 "721.7605590820312",125 "917.9664306640625"126 ]127 },128 "dimension": "SpatialReasoning",129 "subdimension": "VisualGrounding",130 "question_task": "VideoQuestionAnswering",131 "question_format": "ExactMatchQuestion",132 "video_id": "jgepqEjfgPRa",133 "video_url": "https://www.instagram.com/big.blue.sea/reel/C5d2jxbtI-i",134 "video_scene": "000005_000009",135 "id": 1,136 "original_video_id": "C5d2jxbtI-i",137 "turns": [138 {139 "turn_id": 1,140 "id": 1,141 "assistant_response": "[{\"function\": {\"name\": \"object_grounding\", \"arguments\": {\"prompt\": \"A region of a species with long, slender body and dark mottled skin, partially hidden in crevices\", \"ground_type\": \"anything\"}}}]",142 "user_query": "Localize the species with the traits of long, slender body, dark mottled skin, partially hidden in crevices in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",143 "visual_input": [144 "frames/frame_000000.jpg",145 "frames/frame_000001.jpg",146 "frames/frame_000002.jpg",147 "frames/frame_000003.jpg",148 "frames/frame_000004.jpg"149 ],150 "tool_output": "{\"boxes\": [[[\"276.62890625\", \"462.6394348144531\", \"594.9517822265625\", \"703.8656005859375\"]], [[\"222.19586181640625\", \"453.16802978515625\", \"638.389892578125\", \"735.9738159179688\"]], [[\"173.78103637695312\", \"424.33538818359375\", \"666.8064575195312\", \"767.375\"]], [[\"120.7812271118164\", \"418.6991882324219\", \"700.930419921875\", \"816.4493408203125\"]], [[\"74.67980194091797\", \"412.57354736328125\", \"721.7605590820312\", \"917.9664306640625\"]]]}"151 },152 {153 "id": 1,154 "turn_id": 2,155 "user_query": "Localize the species with the traits of long, slender body, dark mottled skin, partially hidden in crevices in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",156 "assistant_response": "<answer>{\"boxes\": [[[\"276.62890625\", \"462.6394348144531\", \"594.9517822265625\", \"703.8656005859375\"]], [[\"222.19586181640625\", \"453.16802978515625\", \"638.389892578125\", \"735.9738159179688\"]], [[\"173.78103637695312\", \"424.33538818359375\", \"666.8064575195312\", \"767.375\"]], [[\"120.7812271118164\", \"418.6991882324219\", \"700.930419921875\", \"816.4493408203125\"]], [[\"74.67980194091797\", \"412.57354736328125\", \"721.7605590820312\", \"917.9664306640625\"]]]}</answer>",157 "visual_input": [158 "frames/frame_000000.jpg",159 "frames/frame_000001.jpg",160 "frames/frame_000002.jpg",161 "frames/frame_000003.jpg",162 "frames/frame_000004.jpg"163 ]164 }165 ]166 },167 {168 "question": "Localize the species with the traits of elongated body with bold black spots on a pale yellow or cream background, no visible fins in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",169 "answer": "{\"boxes\": [[[\"-1.2423348426818848\", \"624.9266967773438\", \"615.1105346679688\", \"1177.085693359375\"]], [[\"-1.0110855102539062\", \"620.0517578125\", \"612.8483276367188\", \"1175.603759765625\"]], [[\"-1.2226366996765137\", \"587.4548950195312\", \"616.49755859375\", \"1206.384765625\"]], [[\"-1.3282513618469238\", \"525.6884155273438\", \"584.835205078125\", \"1249.59228515625\"]], [[\"0.1201629638671875\", \"484.91510009765625\", \"538.58447265625\", \"1279.9498291015625\"]], [[\"0.050811767578125\", \"462.94158935546875\", \"494.8678283691406\", \"1278.4124755859375\"]], [[\"-0.2506256103515625\", \"452.7391357421875\", \"550.1527709960938\", \"1277.9755859375\"]], [[\"-0.13565540313720703\", \"436.213134765625\", \"572.8688354492188\", \"1278.4464111328125\"]], [[\"-0.607144832611084\", \"433.2828063964844\", \"543.8076171875\", \"1277.150634765625\"]], [[\"41.607177734375\", \"416.93475341796875\", \"594.9736938476562\", \"1277.9483642578125\"]], [[\"238.07034301757812\", \"410.4851989746094\", \"650.815673828125\", \"1282.1025390625\"]], [[\"125.98041534423828\", \"416.63714599609375\", \"639.8779296875\", \"1278.7669677734375\"]], [[\"0.0913238525390625\", \"443.375244140625\", \"594.1116333007812\", \"917.930908203125\"]], [[\"-0.06085395812988281\", \"460.8816833496094\", \"596.2171630859375\", \"926.570556640625\"]], [[\"-0.4688286781311035\", \"490.5439758300781\", \"600.1098022460938\", \"1056.00830078125\"]], [[\"-0.5244040489196777\", \"490.61328125\", \"575.8282470703125\", \"1176.625\"]], [[\"1.949000358581543\", \"481.7449951171875\", \"579.1912841796875\", \"1279.425048828125\"]], [[\"69.43672943115234\", \"459.6177978515625\", \"594.6193237304688\", \"1269.918212890625\"]], [[\"-0.5677270889282227\", \"432.0702819824219\", \"607.6826782226562\", \"1087.43994140625\"]], [[\"-0.5068516731262207\", \"412.4757385253906\", \"633.4567260742188\", \"1124.9791259765625\"]], [[\"-0.4217076301574707\", \"412.6748962402344\", \"703.0580444335938\", \"1277.447998046875\"]], [[\"350.5769348144531\", \"419.7126770019531\", \"721.693603515625\", \"814.7193603515625\"]], [[\"334.2128601074219\", \"432.23333740234375\", \"722.7753295898438\", \"839.8339233398438\"]], [[\"301.8483581542969\", \"449.0555419921875\", \"721.6073608398438\", \"938.6730346679688\"]], [[\"261.796142578125\", \"463.2286376953125\", \"633.3703002929688\", \"1086.6668701171875\"]], [[\"242.94227600097656\", \"464.7237854003906\", \"721.4824829101562\", \"770.0191650390625\"]], [[\"218.91969299316406\", \"469.51519775390625\", \"721.9218139648438\", \"628.6827392578125\"]], [[\"166.67047119140625\", \"493.6417236328125\", \"624.4363403320312\", \"675.5191650390625\"]], [[\"55.449886322021484\", \"522.55908203125\", \"520.0747680664062\", \"765.8172607421875\"]], [[\"17.78658676147461\", \"528.0906982421875\", \"542.7763671875\", \"754.356201171875\"]], [[\"239.36160278320312\", \"531.744140625\", \"606.2805786132812\", \"726.940185546875\"]], [[\"324.04132080078125\", \"528.7509155273438\", \"722.7061767578125\", \"790.53955078125\"]], [[\"341.0425720214844\", \"525.0035400390625\", \"721.1552124023438\", \"824.1580810546875\"]], [[\"382.0069580078125\", \"525.9200439453125\", \"721.2713012695312\", \"887.8455810546875\"]], [[\"381.0517272949219\", \"524.9239501953125\", \"722.0556640625\", \"947.2078857421875\"]], [[\"375.060546875\", \"526.5221557617188\", \"722.4854736328125\", \"964.7164306640625\"]], [[\"383.86724853515625\", \"526.1114501953125\", \"720.5159912109375\", \"793.3648681640625\"]], [[\"375.1120300292969\", \"522.7377319335938\", \"720.4620361328125\", \"816.1923217773438\"]], [[\"317.2228088378906\", \"533.1533813476562\", \"721.6868286132812\", \"713.7244262695312\"]], [[\"270.5935363769531\", \"539.0825805664062\", \"721.9449462890625\", \"744.1182861328125\"]], [[\"263.7226257324219\", \"535.5843505859375\", \"721.4373779296875\", \"791.5206298828125\"]], [[\"282.39984130859375\", \"535.236572265625\", \"720.139404296875\", \"826.6901245117188\"]], [[\"286.7592468261719\", \"544.1826171875\", \"722.2698974609375\", \"716.0152587890625\"]], [[\"301.837890625\", \"553.74462890625\", \"721.9244995117188\", \"694.4204711914062\"]], [[\"339.0687561035156\", \"562.4864501953125\", \"722.6193237304688\", \"722.6263427734375\"]], [[\"361.6699523925781\", \"566.44580078125\", \"721.9244995117188\", \"750.076416015625\"]], [[\"384.3092346191406\", \"570.6111450195312\", \"722.527099609375\", \"758.9751586914062\"]], [[\"383.1654357910156\", \"589.1984252929688\", \"723.7759399414062\", \"766.72412109375\"]], [[\"375.9954528808594\", \"600.4859619140625\", \"721.982666015625\", \"701.1436767578125\"]], [[\"380.2048034667969\", \"603.1229248046875\", \"723.0911254882812\", \"715.7564697265625\"]], [[\"395.2603759765625\", \"611.3165283203125\", \"719.6414794921875\", \"726.0472412109375\"]], [[\"396.441650390625\", \"618.9838256835938\", \"721.0667114257812\", \"741.470458984375\"]], [[\"392.41058349609375\", \"620.9131469726562\", \"722.0712890625\", \"756.9170532226562\"]]]}",170 "explanation": {171 "0": [172 "-1.2423348426818848",173 "624.9266967773438",174 "615.1105346679688",175 "1177.085693359375"176 ],177 "1": [178 "-1.0110855102539062",179 "620.0517578125",180 "612.8483276367188",181 "1175.603759765625"182 ],183 "2": [184 "-1.2226366996765137",185 "587.4548950195312",186 "616.49755859375",187 "1206.384765625"188 ],189 "3": [190 "-1.3282513618469238",191 "525.6884155273438",192 "584.835205078125",193 "1249.59228515625"194 ],195 "4": [196 "0.1201629638671875",197 "484.91510009765625",198 "538.58447265625",199 "1279.9498291015625"200 ],201 "5": [202 "0.050811767578125",203 "462.94158935546875",204 "494.8678283691406",205 "1278.4124755859375"206 ],207 "6": [208 "-0.2506256103515625",209 "452.7391357421875",210 "550.1527709960938",211 "1277.9755859375"212 ],213 "7": [214 "-0.13565540313720703",215 "436.213134765625",216 "572.8688354492188",217 "1278.4464111328125"218 ],219 "8": [220 "-0.607144832611084",221 "433.2828063964844",222 "543.8076171875",223 "1277.150634765625"224 ],225 "9": [226 "41.607177734375",227 "416.93475341796875",228 "594.9736938476562",229 "1277.9483642578125"230 ],231 "10": [232 "238.07034301757812",233 "410.4851989746094",234 "650.815673828125",235 "1282.1025390625"236 ],237 "11": [238 "125.98041534423828",239 "416.63714599609375",240 "639.8779296875",241 "1278.7669677734375"242 ],243 "12": [244 "0.0913238525390625",245 "443.375244140625",246 "594.1116333007812",247 "917.930908203125"248 ],249 "13": [250 "-0.06085395812988281",251 "460.8816833496094",252 "596.2171630859375",253 "926.570556640625"254 ],255 "14": [256 "-0.4688286781311035",257 "490.5439758300781",258 "600.1098022460938",259 "1056.00830078125"260 ],261 "15": [262 "-0.5244040489196777",263 "490.61328125",264 "575.8282470703125",265 "1176.625"266 ],267 "16": [268 "1.949000358581543",269 "481.7449951171875",270 "579.1912841796875",271 "1279.425048828125"272 ],273 "17": [274 "69.43672943115234",275 "459.6177978515625",276 "594.6193237304688",277 "1269.918212890625"278 ],279 "18": [280 "-0.5677270889282227",281 "432.0702819824219",282 "607.6826782226562",283 "1087.43994140625"284 ],285 "19": [286 "-0.5068516731262207",287 "412.4757385253906",288 "633.4567260742188",289 "1124.9791259765625"290 ],291 "20": [292 "-0.4217076301574707",293 "412.6748962402344",294 "703.0580444335938",295 "1277.447998046875"296 ],297 "21": [298 "350.5769348144531",299 "419.7126770019531",300 "721.693603515625",301 "814.7193603515625"302 ],303 "22": [304 "334.2128601074219",305 "432.23333740234375",306 "722.7753295898438",307 "839.8339233398438"308 ],309 "23": [310 "301.8483581542969",311 "449.0555419921875",312 "721.6073608398438",313 "938.6730346679688"314 ],315 "24": [316 "261.796142578125",317 "463.2286376953125",318 "633.3703002929688",319 "1086.6668701171875"320 ],321 "25": [322 "242.94227600097656",323 "464.7237854003906",324 "721.4824829101562",325 "770.0191650390625"326 ],327 "26": [328 "218.91969299316406",329 "469.51519775390625",330 "721.9218139648438",331 "628.6827392578125"332 ],333 "27": [334 "166.67047119140625",335 "493.6417236328125",336 "624.4363403320312",337 "675.5191650390625"338 ],339 "28": [340 "55.449886322021484",341 "522.55908203125",342 "520.0747680664062",343 "765.8172607421875"344 ],345 "29": [346 "17.78658676147461",347 "528.0906982421875",348 "542.7763671875",349 "754.356201171875"350 ],351 "30": [352 "239.36160278320312",353 "531.744140625",354 "606.2805786132812",355 "726.940185546875"356 ],357 "31": [358 "324.04132080078125",359 "528.7509155273438",360 "722.7061767578125",361 "790.53955078125"362 ],363 "32": [364 "341.0425720214844",365 "525.0035400390625",366 "721.1552124023438",367 "824.1580810546875"368 ],369 "33": [370 "382.0069580078125",371 "525.9200439453125",372 "721.2713012695312",373 "887.8455810546875"374 ],375 "34": [376 "381.0517272949219",377 "524.9239501953125",378 "722.0556640625",379 "947.2078857421875"380 ],381 "35": [382 "375.060546875",383 "526.5221557617188",384 "722.4854736328125",385 "964.7164306640625"386 ],387 "36": [388 "383.86724853515625",389 "526.1114501953125",390 "720.5159912109375",391 "793.3648681640625"392 ],393 "37": [394 "375.1120300292969",395 "522.7377319335938",396 "720.4620361328125",397 "816.1923217773438"398 ],399 "38": [400 "317.2228088378906",401 "533.1533813476562",402 "721.6868286132812",403 "713.7244262695312"404 ],405 "39": [406 "270.5935363769531",407 "539.0825805664062",408 "721.9449462890625",409 "744.1182861328125"410 ],411 "40": [412 "263.7226257324219",413 "535.5843505859375",414 "721.4373779296875",415 "791.5206298828125"416 ],417 "41": [418 "282.39984130859375",419 "535.236572265625",420 "720.139404296875",421 "826.6901245117188"422 ],423 "42": [424 "286.7592468261719",425 "544.1826171875",426 "722.2698974609375",427 "716.0152587890625"428 ],429 "43": [430 "301.837890625",431 "553.74462890625",432 "721.9244995117188",433 "694.4204711914062"434 ],435 "44": [436 "339.0687561035156",437 "562.4864501953125",438 "722.6193237304688",439 "722.6263427734375"440 ],441 "45": [442 "361.6699523925781",443 "566.44580078125",444 "721.9244995117188",445 "750.076416015625"446 ],447 "46": [448 "384.3092346191406",449 "570.6111450195312",450 "722.527099609375",451 "758.9751586914062"452 ],453 "47": [454 "383.1654357910156",455 "589.1984252929688",456 "723.7759399414062",457 "766.72412109375"458 ],459 "48": [460 "375.9954528808594",461 "600.4859619140625",462 "721.982666015625",463 "701.1436767578125"464 ],465 "49": [466 "380.2048034667969",467 "603.1229248046875",468 "723.0911254882812",469 "715.7564697265625"470 ],471 "50": [472 "395.2603759765625",473 "611.3165283203125",474 "719.6414794921875",475 "726.0472412109375"476 ],477 "51": [478 "396.441650390625",479 "618.9838256835938",480 "721.0667114257812",481 "741.470458984375"482 ],483 "52": [484 "392.41058349609375",485 "620.9131469726562",486 "722.0712890625",487 "756.9170532226562"488 ]489 },490 "dimension": "SpatialReasoning",491 "subdimension": "VisualGrounding",492 "question_task": "VideoQuestionAnswering",493 "question_format": "ExactMatchQuestion",494 "video_id": "JevZrvHEDM7z",495 "video_url": "https://www.instagram.com/big.blue.sea/reel/DMVpvTqt4Ex",496 "video_scene": "000000_000052",497 "id": 2,498 "original_video_id": "DMVpvTqt4Ex",499 "turns": [500 {501 "turn_id": 1,502 "id": 2,503 "assistant_response": "[{\"function\": {\"name\": \"object_grounding\", \"arguments\": {\"prompt\": \"A region of a species with an elongated body, bold black spots on a pale yellow or cream background, and no visible fins\", \"ground_type\": \"anything\"}}}]",504 "user_query": "Localize the species with the traits of elongated body with bold black spots on a pale yellow or cream background, no visible fins in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",505 "visual_input": [506 "frames/frame_000000.jpg",507 "frames/frame_000001.jpg",508 "frames/frame_000002.jpg",509 "frames/frame_000003.jpg",510 "frames/frame_000004.jpg",511 "frames/frame_000005.jpg",512 "frames/frame_000006.jpg",513 "frames/frame_000007.jpg",514 "frames/frame_000008.jpg",515 "frames/frame_000009.jpg",516 "frames/frame_000010.jpg",517 "frames/frame_000011.jpg",518 "frames/frame_000012.jpg",519 "frames/frame_000013.jpg",520 "frames/frame_000014.jpg",521 "frames/frame_000015.jpg",522 "frames/frame_000016.jpg",523 "frames/frame_000017.jpg",524 "frames/frame_000018.jpg",525 "frames/frame_000019.jpg",526 "frames/frame_000020.jpg",527 "frames/frame_000021.jpg",528 "frames/frame_000022.jpg",529 "frames/frame_000023.jpg",530 "frames/frame_000024.jpg",531 "frames/frame_000025.jpg",532 "frames/frame_000026.jpg",533 "frames/frame_000027.jpg",534 "frames/frame_000028.jpg",535 "frames/frame_000029.jpg",536 "frames/frame_000030.jpg",537 "frames/frame_000031.jpg",538 "frames/frame_000032.jpg",539 "frames/frame_000033.jpg",540 "frames/frame_000034.jpg",541 "frames/frame_000035.jpg",542 "frames/frame_000036.jpg",543 "frames/frame_000037.jpg",544 "frames/frame_000038.jpg",545 "frames/frame_000039.jpg",546 "frames/frame_000040.jpg",547 "frames/frame_000041.jpg",548 "frames/frame_000042.jpg",549 "frames/frame_000043.jpg",550 "frames/frame_000044.jpg",551 "frames/frame_000045.jpg",552 "frames/frame_000046.jpg",553 "frames/frame_000047.jpg",554 "frames/frame_000048.jpg",555 "frames/frame_000049.jpg",556 "frames/frame_000050.jpg",557 "frames/frame_000051.jpg",558 "frames/frame_000052.jpg"559 ],560 "tool_output": "{\"boxes\": [[[\"-1.2423348426818848\", \"624.9266967773438\", \"615.1105346679688\", \"1177.085693359375\"]], [[\"-1.0110855102539062\", \"620.0517578125\", \"612.8483276367188\", \"1175.603759765625\"]], [[\"-1.2226366996765137\", \"587.4548950195312\", \"616.49755859375\", \"1206.384765625\"]], [[\"-1.3282513618469238\", \"525.6884155273438\", \"584.835205078125\", \"1249.59228515625\"]], [[\"0.1201629638671875\", \"484.91510009765625\", \"538.58447265625\", \"1279.9498291015625\"]], [[\"0.050811767578125\", \"462.94158935546875\", \"494.8678283691406\", \"1278.4124755859375\"]], [[\"-0.2506256103515625\", \"452.7391357421875\", \"550.1527709960938\", \"1277.9755859375\"]], [[\"-0.13565540313720703\", \"436.213134765625\", \"572.8688354492188\", \"1278.4464111328125\"]], [[\"-0.607144832611084\", \"433.2828063964844\", \"543.8076171875\", \"1277.150634765625\"]], [[\"41.607177734375\", \"416.93475341796875\", \"594.9736938476562\", \"1277.9483642578125\"]], [[\"238.07034301757812\", \"410.4851989746094\", \"650.815673828125\", \"1282.1025390625\"]], [[\"125.98041534423828\", \"416.63714599609375\", \"639.8779296875\", \"1278.7669677734375\"]], [[\"0.0913238525390625\", \"443.375244140625\", \"594.1116333007812\", \"917.930908203125\"]], [[\"-0.06085395812988281\", \"460.8816833496094\", \"596.2171630859375\", \"926.570556640625\"]], [[\"-0.4688286781311035\", \"490.5439758300781\", \"600.1098022460938\", \"1056.00830078125\"]], [[\"-0.5244040489196777\", \"490.61328125\", \"575.8282470703125\", \"1176.625\"]], [[\"1.949000358581543\", \"481.7449951171875\", \"579.1912841796875\", \"1279.425048828125\"]], [[\"69.43672943115234\", \"459.6177978515625\", \"594.6193237304688\", \"1269.918212890625\"]], [[\"-0.5677270889282227\", \"432.0702819824219\", \"607.6826782226562\", \"1087.43994140625\"]], [[\"-0.5068516731262207\", \"412.4757385253906\", \"633.4567260742188\", \"1124.9791259765625\"]], [[\"-0.4217076301574707\", \"412.6748962402344\", \"703.0580444335938\", \"1277.447998046875\"]], [[\"350.5769348144531\", \"419.7126770019531\", \"721.693603515625\", \"814.7193603515625\"]], [[\"334.2128601074219\", \"432.23333740234375\", \"722.7753295898438\", \"839.8339233398438\"]], [[\"301.8483581542969\", \"449.0555419921875\", \"721.6073608398438\", \"938.6730346679688\"]], [[\"261.796142578125\", \"463.2286376953125\", \"633.3703002929688\", \"1086.6668701171875\"]], [[\"242.94227600097656\", \"464.7237854003906\", \"721.4824829101562\", \"770.0191650390625\"]], [[\"218.91969299316406\", \"469.51519775390625\", \"721.9218139648438\", \"628.6827392578125\"]], [[\"166.67047119140625\", \"493.6417236328125\", \"624.4363403320312\", \"675.5191650390625\"]], [[\"55.449886322021484\", \"522.55908203125\", \"520.0747680664062\", \"765.8172607421875\"]], [[\"17.78658676147461\", \"528.0906982421875\", \"542.7763671875\", \"754.356201171875\"]], [[\"239.36160278320312\", \"531.744140625\", \"606.2805786132812\", \"726.940185546875\"]], [[\"324.04132080078125\", \"528.7509155273438\", \"722.7061767578125\", \"790.53955078125\"]], [[\"341.0425720214844\", \"525.0035400390625\", \"721.1552124023438\", \"824.1580810546875\"]], [[\"382.0069580078125\", \"525.9200439453125\", \"721.2713012695312\", \"887.8455810546875\"]], [[\"381.0517272949219\", \"524.9239501953125\", \"722.0556640625\", \"947.2078857421875\"]], [[\"375.060546875\", \"526.5221557617188\", \"722.4854736328125\", \"964.7164306640625\"]], [[\"383.86724853515625\", \"526.1114501953125\", \"720.5159912109375\", \"793.3648681640625\"]], [[\"375.1120300292969\", \"522.7377319335938\", \"720.4620361328125\", \"816.1923217773438\"]], [[\"317.2228088378906\", \"533.1533813476562\", \"721.6868286132812\", \"713.7244262695312\"]], [[\"270.5935363769531\", \"539.0825805664062\", \"721.9449462890625\", \"744.1182861328125\"]], [[\"263.7226257324219\", \"535.5843505859375\", \"721.4373779296875\", \"791.5206298828125\"]], [[\"282.39984130859375\", \"535.236572265625\", \"720.139404296875\", \"826.6901245117188\"]], [[\"286.7592468261719\", \"544.1826171875\", \"722.2698974609375\", \"716.0152587890625\"]], [[\"301.837890625\", \"553.74462890625\", \"721.9244995117188\", \"694.4204711914062\"]], [[\"339.0687561035156\", \"562.4864501953125\", \"722.6193237304688\", \"722.6263427734375\"]], [[\"361.6699523925781\", \"566.44580078125\", \"721.9244995117188\", \"750.076416015625\"]], [[\"384.3092346191406\", \"570.6111450195312\", \"722.527099609375\", \"758.9751586914062\"]], [[\"383.1654357910156\", \"589.1984252929688\", \"723.7759399414062\", \"766.72412109375\"]], [[\"375.9954528808594\", \"600.4859619140625\", \"721.982666015625\", \"701.1436767578125\"]], [[\"380.2048034667969\", \"603.1229248046875\", \"723.0911254882812\", \"715.7564697265625\"]], [[\"395.2603759765625\", \"611.3165283203125\", \"719.6414794921875\", \"726.0472412109375\"]], [[\"396.441650390625\", \"618.9838256835938\", \"721.0667114257812\", \"741.470458984375\"]], [[\"392.41058349609375\", \"620.9131469726562\", \"722.0712890625\", \"756.9170532226562\"]]]}"561 },562 {563 "id": 2,564 "turn_id": 2,565 "user_query": "Localize the species with the traits of elongated body with bold black spots on a pale yellow or cream background, no visible fins in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",566 "assistant_response": "<answer>{\"boxes\": [[[\"-1.2423348426818848\", \"624.9266967773438\", \"615.1105346679688\", \"1177.085693359375\"]], [[\"-1.0110855102539062\", \"620.0517578125\", \"612.8483276367188\", \"1175.603759765625\"]], [[\"-1.2226366996765137\", \"587.4548950195312\", \"616.49755859375\", \"1206.384765625\"]], [[\"-1.3282513618469238\", \"525.6884155273438\", \"584.835205078125\", \"1249.59228515625\"]], [[\"0.1201629638671875\", \"484.91510009765625\", \"538.58447265625\", \"1279.9498291015625\"]], [[\"0.050811767578125\", \"462.94158935546875\", \"494.8678283691406\", \"1278.4124755859375\"]], [[\"-0.2506256103515625\", \"452.7391357421875\", \"550.1527709960938\", \"1277.9755859375\"]], [[\"-0.13565540313720703\", \"436.213134765625\", \"572.8688354492188\", \"1278.4464111328125\"]], [[\"-0.607144832611084\", \"433.2828063964844\", \"543.8076171875\", \"1277.150634765625\"]], [[\"41.607177734375\", \"416.93475341796875\", \"594.9736938476562\", \"1277.9483642578125\"]], [[\"238.07034301757812\", \"410.4851989746094\", \"650.815673828125\", \"1282.1025390625\"]], [[\"125.98041534423828\", \"416.63714599609375\", \"639.8779296875\", \"1278.7669677734375\"]], [[\"0.0913238525390625\", \"443.375244140625\", \"594.1116333007812\", \"917.930908203125\"]], [[\"-0.06085395812988281\", \"460.8816833496094\", \"596.2171630859375\", \"926.570556640625\"]], [[\"-0.4688286781311035\", \"490.5439758300781\", \"600.1098022460938\", \"1056.00830078125\"]], [[\"-0.5244040489196777\", \"490.61328125\", \"575.8282470703125\", \"1176.625\"]], [[\"1.949000358581543\", \"481.7449951171875\", \"579.1912841796875\", \"1279.425048828125\"]], [[\"69.43672943115234\", \"459.6177978515625\", \"594.6193237304688\", \"1269.918212890625\"]], [[\"-0.5677270889282227\", \"432.0702819824219\", \"607.6826782226562\", \"1087.43994140625\"]], [[\"-0.5068516731262207\", \"412.4757385253906\", \"633.4567260742188\", \"1124.9791259765625\"]], [[\"-0.4217076301574707\", \"412.6748962402344\", \"703.0580444335938\", \"1277.447998046875\"]], [[\"350.5769348144531\", \"419.7126770019531\", \"721.693603515625\", \"814.7193603515625\"]], [[\"334.2128601074219\", \"432.23333740234375\", \"722.7753295898438\", \"839.8339233398438\"]], [[\"301.8483581542969\", \"449.0555419921875\", \"721.6073608398438\", \"938.6730346679688\"]], [[\"261.796142578125\", \"463.2286376953125\", \"633.3703002929688\", \"1086.6668701171875\"]], [[\"242.94227600097656\", \"464.7237854003906\", \"721.4824829101562\", \"770.0191650390625\"]], [[\"218.91969299316406\", \"469.51519775390625\", \"721.9218139648438\", \"628.6827392578125\"]], [[\"166.67047119140625\", \"493.6417236328125\", \"624.4363403320312\", \"675.5191650390625\"]], [[\"55.449886322021484\", \"522.55908203125\", \"520.0747680664062\", \"765.8172607421875\"]], [[\"17.78658676147461\", \"528.0906982421875\", \"542.7763671875\", \"754.356201171875\"]], [[\"239.36160278320312\", \"531.744140625\", \"606.2805786132812\", \"726.940185546875\"]], [[\"324.04132080078125\", \"528.7509155273438\", \"722.7061767578125\", \"790.53955078125\"]], [[\"341.0425720214844\", \"525.0035400390625\", \"721.1552124023438\", \"824.1580810546875\"]], [[\"382.0069580078125\", \"525.9200439453125\", \"721.2713012695312\", \"887.8455810546875\"]], [[\"381.0517272949219\", \"524.9239501953125\", \"722.0556640625\", \"947.2078857421875\"]], [[\"375.060546875\", \"526.5221557617188\", \"722.4854736328125\", \"964.7164306640625\"]], [[\"383.86724853515625\", \"526.1114501953125\", \"720.5159912109375\", \"793.3648681640625\"]], [[\"375.1120300292969\", \"522.7377319335938\", \"720.4620361328125\", \"816.1923217773438\"]], [[\"317.2228088378906\", \"533.1533813476562\", \"721.6868286132812\", \"713.7244262695312\"]], [[\"270.5935363769531\", \"539.0825805664062\", \"721.9449462890625\", \"744.1182861328125\"]], [[\"263.7226257324219\", \"535.5843505859375\", \"721.4373779296875\", \"791.5206298828125\"]], [[\"282.39984130859375\", \"535.236572265625\", \"720.139404296875\", \"826.6901245117188\"]], [[\"286.7592468261719\", \"544.1826171875\", \"722.2698974609375\", \"716.0152587890625\"]], [[\"301.837890625\", \"553.74462890625\", \"721.9244995117188\", \"694.4204711914062\"]], [[\"339.0687561035156\", \"562.4864501953125\", \"722.6193237304688\", \"722.6263427734375\"]], [[\"361.6699523925781\", \"566.44580078125\", \"721.9244995117188\", \"750.076416015625\"]], [[\"384.3092346191406\", \"570.6111450195312\", \"722.527099609375\", \"758.9751586914062\"]], [[\"383.1654357910156\", \"589.1984252929688\", \"723.7759399414062\", \"766.72412109375\"]], [[\"375.9954528808594\", \"600.4859619140625\", \"721.982666015625\", \"701.1436767578125\"]], [[\"380.2048034667969\", \"603.1229248046875\", \"723.0911254882812\", \"715.7564697265625\"]], [[\"395.2603759765625\", \"611.3165283203125\", \"719.6414794921875\", \"726.0472412109375\"]], [[\"396.441650390625\", \"618.9838256835938\", \"721.0667114257812\", \"741.470458984375\"]], [[\"392.41058349609375\", \"620.9131469726562\", \"722.0712890625\", \"756.9170532226562\"]]]}</answer>",567 "visual_input": [568 "frames/frame_000000.jpg",569 "frames/frame_000001.jpg",570 "frames/frame_000002.jpg",571 "frames/frame_000003.jpg",572 "frames/frame_000004.jpg",573 "frames/frame_000005.jpg",574 "frames/frame_000006.jpg",575 "frames/frame_000007.jpg",576 "frames/frame_000008.jpg",577 "frames/frame_000009.jpg",578 "frames/frame_000010.jpg",579 "frames/frame_000011.jpg",580 "frames/frame_000012.jpg",581 "frames/frame_000013.jpg",582 "frames/frame_000014.jpg",583 "frames/frame_000015.jpg",584 "frames/frame_000016.jpg",585 "frames/frame_000017.jpg",586 "frames/frame_000018.jpg",587 "frames/frame_000019.jpg",588 "frames/frame_000020.jpg",589 "frames/frame_000021.jpg",590 "frames/frame_000022.jpg",591 "frames/frame_000023.jpg",592 "frames/frame_000024.jpg",593 "frames/frame_000025.jpg",594 "frames/frame_000026.jpg",595 "frames/frame_000027.jpg",596 "frames/frame_000028.jpg",597 "frames/frame_000029.jpg",598 "frames/frame_000030.jpg",599 "frames/frame_000031.jpg",600 "frames/frame_000032.jpg",601 "frames/frame_000033.jpg",602 "frames/frame_000034.jpg",603 "frames/frame_000035.jpg",604 "frames/frame_000036.jpg",605 "frames/frame_000037.jpg",606 "frames/frame_000038.jpg",607 "frames/frame_000039.jpg",608 "frames/frame_000040.jpg",609 "frames/frame_000041.jpg",610 "frames/frame_000042.jpg",611 "frames/frame_000043.jpg",612 "frames/frame_000044.jpg",613 "frames/frame_000045.jpg",614 "frames/frame_000046.jpg",615 "frames/frame_000047.jpg",616 "frames/frame_000048.jpg",617 "frames/frame_000049.jpg",618 "frames/frame_000050.jpg",619 "frames/frame_000051.jpg",620 "frames/frame_000052.jpg"621 ]622 }623 ]624 },625 {626 "question": "Localize the species with the traits of grayish body with yellow tail, beak-like mouth in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",627 "answer": "{\"boxes\": [[[\"21.99608612060547\", \"502.0540771484375\", \"83.3436050415039\", \"761.8826904296875\"]], [[\"-1.0257303714752197\", \"498.461669921875\", \"77.67058563232422\", \"782.1978149414062\"]], [[\"0.9568995237350464\", \"490.3488464355469\", \"65.39704895019531\", \"775.7484130859375\"]]]}",628 "explanation": {629 "0": [630 "21.99608612060547",631 "502.0540771484375",632 "83.3436050415039",633 "761.8826904296875"634 ],635 "1": [636 "-1.0257303714752197",637 "498.461669921875",638 "77.67058563232422",639 "782.1978149414062"640 ],641 "2": [642 "0.9568995237350464",643 "490.3488464355469",644 "65.39704895019531",645 "775.7484130859375"646 ]647 },648 "dimension": "SpatialReasoning",649 "subdimension": "VisualGrounding",650 "question_task": "VideoQuestionAnswering",651 "question_format": "ExactMatchQuestion",652 "video_id": "K6eWyCiJ9XpY",653 "video_url": "https://www.instagram.com/evgenii_dvorkin/reel/C9h5fgwthnA",654 "video_scene": "000018_000020",655 "id": 3,656 "original_video_id": "C9h5fgwthnA",657 "turns": [658 {659 "turn_id": 1,660 "id": 3,661 "assistant_response": "[{\"function\": {\"name\": \"object_grounding\", \"arguments\": {\"prompt\": \"A region of a species with grayish body and yellow tail, featuring a beak-like mouth\", \"ground_type\": \"anything\"}}}]",662 "user_query": "Localize the species with the traits of grayish body with yellow tail, beak-like mouth in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",663 "visual_input": [664 "frames/frame_000000.jpg",665 "frames/frame_000001.jpg",666 "frames/frame_000002.jpg"667 ],668 "tool_output": "{\"boxes\": [[[\"21.99608612060547\", \"502.0540771484375\", \"83.3436050415039\", \"761.8826904296875\"]], [[\"-1.0257303714752197\", \"498.461669921875\", \"77.67058563232422\", \"782.1978149414062\"]], [[\"0.9568995237350464\", \"490.3488464355469\", \"65.39704895019531\", \"775.7484130859375\"]]]}"669 },670 {671 "id": 3,672 "turn_id": 2,673 "user_query": "Localize the species with the traits of grayish body with yellow tail, beak-like mouth in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",674 "assistant_response": "<answer>{\"boxes\": [[[\"21.99608612060547\", \"502.0540771484375\", \"83.3436050415039\", \"761.8826904296875\"]], [[\"-1.0257303714752197\", \"498.461669921875\", \"77.67058563232422\", \"782.1978149414062\"]], [[\"0.9568995237350464\", \"490.3488464355469\", \"65.39704895019531\", \"775.7484130859375\"]]]}</answer>",675 "visual_input": [676 "frames/frame_000000.jpg",677 "frames/frame_000001.jpg",678 "frames/frame_000002.jpg"679 ]680 }681 ]682 },683 {684 "question": "Localize the species with the traits of dark blue body, distinctive dorsal fin, elongated tail, flat body shape in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",685 "answer": "{\"boxes\": [[[\"414.59906005859375\", \"613.7180786132812\", \"551.5039672851562\", \"783.8037719726562\"]], [[\"359.32684326171875\", \"573.611083984375\", \"612.4398803710938\", \"812.6080932617188\"]], [[\"315.8789367675781\", \"504.1756591796875\", \"594.52783203125\", \"792.0128784179688\"]], [[\"296.7608947753906\", \"406.1106872558594\", \"558.7424926757812\", \"773.2787475585938\"]], [[\"311.3494567871094\", \"347.7359619140625\", \"574.1715087890625\", \"808.6477661132812\"]], [[\"244.10659790039062\", \"294.1944580078125\", \"616.5473022460938\", \"808.168212890625\"]], [[\"203.639404296875\", \"221.08001708984375\", \"597.6879272460938\", \"798.3822021484375\"]], [[\"325.7801513671875\", \"207.44186401367188\", \"622.8536376953125\", \"812.4057006835938\"]], [[\"344.8687744140625\", \"281.1687927246094\", \"625.1732788085938\", \"844.4056396484375\"]], [[\"232.54095458984375\", \"300.853271484375\", \"609.5475463867188\", \"888.765380859375\"]], [[\"106.53135681152344\", \"226.45108032226562\", \"691.7739868164062\", \"912.6258544921875\"]], [[\"52.388648986816406\", \"164.33505249023438\", \"549.0205688476562\", \"880.557861328125\"]]]}",686 "explanation": {687 "0": [688 "414.59906005859375",689 "613.7180786132812",690 "551.5039672851562",691 "783.8037719726562"692 ],693 "1": [694 "359.32684326171875",695 "573.611083984375",696 "612.4398803710938",697 "812.6080932617188"698 ],699 "2": [700 "315.8789367675781",701 "504.1756591796875",702 "594.52783203125",703 "792.0128784179688"704 ],705 "3": [706 "296.7608947753906",707 "406.1106872558594",708 "558.7424926757812",709 "773.2787475585938"710 ],711 "4": [712 "311.3494567871094",713 "347.7359619140625",714 "574.1715087890625",715 "808.6477661132812"716 ],717 "5": [718 "244.10659790039062",719 "294.1944580078125",720 "616.5473022460938",721 "808.168212890625"722 ],723 "6": [724 "203.639404296875",725 "221.08001708984375",726 "597.6879272460938",727 "798.3822021484375"728 ],729 "7": [730 "325.7801513671875",731 "207.44186401367188",732 "622.8536376953125",733 "812.4057006835938"734 ],735 "8": [736 "344.8687744140625",737 "281.1687927246094",738 "625.1732788085938",739 "844.4056396484375"740 ],741 "9": [742 "232.54095458984375",743 "300.853271484375",744 "609.5475463867188",745 "888.765380859375"746 ],747 "10": [748 "106.53135681152344",749 "226.45108032226562",750 "691.7739868164062",751 "912.6258544921875"752 ],753 "11": [754 "52.388648986816406",755 "164.33505249023438",756 "549.0205688476562",757 "880.557861328125"758 ]759 },760 "dimension": "SpatialReasoning",761 "subdimension": "VisualGrounding",762 "question_task": "VideoQuestionAnswering",763 "question_format": "ExactMatchQuestion",764 "video_id": "pxjaawHQdyCl",765 "video_url": "https://www.instagram.com/big.blue.sea/reel/CqmCRG_LVf7",766 "video_scene": "000000_000011",767 "id": 4,768 "original_video_id": "CqmCRG_LVf7",769 "turns": [770 {771 "turn_id": 1,772 "id": 4,773 "assistant_response": "[{\"function\": {\"name\": \"object_grounding\", \"arguments\": {\"prompt\": \"A region of a fish with a dark blue body, distinctive dorsal fin, elongated tail, and flat body shape\", \"ground_type\": \"anything\"}}}]",774 "user_query": "Localize the species with the traits of dark blue body, distinctive dorsal fin, elongated tail, flat body shape in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",775 "visual_input": [776 "frames/frame_000000.jpg",777 "frames/frame_000001.jpg",778 "frames/frame_000002.jpg",779 "frames/frame_000003.jpg",780 "frames/frame_000004.jpg",781 "frames/frame_000005.jpg",782 "frames/frame_000006.jpg",783 "frames/frame_000007.jpg",784 "frames/frame_000008.jpg",785 "frames/frame_000009.jpg",786 "frames/frame_000010.jpg",787 "frames/frame_000011.jpg"788 ],789 "tool_output": "{\"boxes\": [[[\"414.59906005859375\", \"613.7180786132812\", \"551.5039672851562\", \"783.8037719726562\"]], [[\"359.32684326171875\", \"573.611083984375\", \"612.4398803710938\", \"812.6080932617188\"]], [[\"315.8789367675781\", \"504.1756591796875\", \"594.52783203125\", \"792.0128784179688\"]], [[\"296.7608947753906\", \"406.1106872558594\", \"558.7424926757812\", \"773.2787475585938\"]], [[\"311.3494567871094\", \"347.7359619140625\", \"574.1715087890625\", \"808.6477661132812\"]], [[\"244.10659790039062\", \"294.1944580078125\", \"616.5473022460938\", \"808.168212890625\"]], [[\"203.639404296875\", \"221.08001708984375\", \"597.6879272460938\", \"798.3822021484375\"]], [[\"325.7801513671875\", \"207.44186401367188\", \"622.8536376953125\", \"812.4057006835938\"]], [[\"344.8687744140625\", \"281.1687927246094\", \"625.1732788085938\", \"844.4056396484375\"]], [[\"232.54095458984375\", \"300.853271484375\", \"609.5475463867188\", \"888.765380859375\"]], [[\"106.53135681152344\", \"226.45108032226562\", \"691.7739868164062\", \"912.6258544921875\"]], [[\"52.388648986816406\", \"164.33505249023438\", \"549.0205688476562\", \"880.557861328125\"]]]}"790 },791 {792 "id": 4,793 "turn_id": 2,794 "user_query": "Localize the species with the traits of dark blue body, distinctive dorsal fin, elongated tail, flat body shape in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",795 "assistant_response": "<answer>{\"boxes\": [[[\"414.59906005859375\", \"613.7180786132812\", \"551.5039672851562\", \"783.8037719726562\"]], [[\"359.32684326171875\", \"573.611083984375\", \"612.4398803710938\", \"812.6080932617188\"]], [[\"315.8789367675781\", \"504.1756591796875\", \"594.52783203125\", \"792.0128784179688\"]], [[\"296.7608947753906\", \"406.1106872558594\", \"558.7424926757812\", \"773.2787475585938\"]], [[\"311.3494567871094\", \"347.7359619140625\", \"574.1715087890625\", \"808.6477661132812\"]], [[\"244.10659790039062\", \"294.1944580078125\", \"616.5473022460938\", \"808.168212890625\"]], [[\"203.639404296875\", \"221.08001708984375\", \"597.6879272460938\", \"798.3822021484375\"]], [[\"325.7801513671875\", \"207.44186401367188\", \"622.8536376953125\", \"812.4057006835938\"]], [[\"344.8687744140625\", \"281.1687927246094\", \"625.1732788085938\", \"844.4056396484375\"]], [[\"232.54095458984375\", \"300.853271484375\", \"609.5475463867188\", \"888.765380859375\"]], [[\"106.53135681152344\", \"226.45108032226562\", \"691.7739868164062\", \"912.6258544921875\"]], [[\"52.388648986816406\", \"164.33505249023438\", \"549.0205688476562\", \"880.557861328125\"]]]}</answer>",796 "visual_input": [797 "frames/frame_000000.jpg",798 "frames/frame_000001.jpg",799 "frames/frame_000002.jpg",800 "frames/frame_000003.jpg",801 "frames/frame_000004.jpg",802 "frames/frame_000005.jpg",803 "frames/frame_000006.jpg",804 "frames/frame_000007.jpg",805 "frames/frame_000008.jpg",806 "frames/frame_000009.jpg",807 "frames/frame_000010.jpg",808 "frames/frame_000011.jpg"809 ]810 }811 ]812 },813 {814 "question": "Localize the species with the traits of brownish-red body with white speckles, large eyes, tentacles visible, textured skin in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",815 "answer": "{\"boxes\": [[[\"60.52213668823242\", \"275.92340087890625\", \"724.6060791015625\", \"592.4597778320312\"]], [[\"39.84159851074219\", \"262.70172119140625\", \"723.0554809570312\", \"568.594482421875\"]], [[\"79.69405364990234\", \"243.9294891357422\", \"726.3303833007812\", \"547.327392578125\"]], [[\"61.52854537963867\", \"329.4207763671875\", \"548.9110107421875\", \"508.7867126464844\"]], [[\"31.69281005859375\", \"325.5604248046875\", \"538.5624389648438\", \"518.9894409179688\"]], [[\"28.702146530151367\", \"348.4562072753906\", \"555.9247436523438\", \"548.6229248046875\"]]]}",816 "explanation": {817 "0": [818 "60.52213668823242",819 "275.92340087890625",820 "724.6060791015625",821 "592.4597778320312"822 ],823 "1": [824 "39.84159851074219",825 "262.70172119140625",826 "723.0554809570312",827 "568.594482421875"828 ],829 "2": [830 "79.69405364990234",831 "243.9294891357422",832 "726.3303833007812",833 "547.327392578125"834 ],835 "3": [836 "61.52854537963867",837 "329.4207763671875",838 "548.9110107421875",839 "508.7867126464844"840 ],841 "4": [842 "31.69281005859375",843 "325.5604248046875",844 "538.5624389648438",845 "518.9894409179688"846 ],847 "5": [848 "28.702146530151367",849 "348.4562072753906",850 "555.9247436523438",851 "548.6229248046875"852 ]853 },854 "dimension": "SpatialReasoning",855 "subdimension": "VisualGrounding",856 "question_task": "VideoQuestionAnswering",857 "question_format": "ExactMatchQuestion",858 "video_id": "PTAb2XNrMQiH",859 "video_url": "https://www.instagram.com/tomparkfilms/reel/CAEKlLLAVT_",860 "video_scene": "000000_000005",861 "id": 5,862 "original_video_id": "CAEKlLLAVT_",863 "turns": [864 {865 "turn_id": 1,866 "id": 5,867 "assistant_response": "[{\"function\": {\"name\": \"object_grounding\", \"arguments\": {\"prompt\": \"A region of a species with brownish-red body, white speckles, large eyes, visible tentacles, and textured skin\", \"ground_type\": \"anything\"}}}]",868 "user_query": "Localize the species with the traits of brownish-red body with white speckles, large eyes, tentacles visible, textured skin in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",869 "visual_input": [870 "frames/frame_000000.jpg",871 "frames/frame_000001.jpg",872 "frames/frame_000002.jpg",873 "frames/frame_000003.jpg",874 "frames/frame_000004.jpg",875 "frames/frame_000005.jpg"876 ],877 "tool_output": "{\"boxes\": [[[\"60.52213668823242\", \"275.92340087890625\", \"724.6060791015625\", \"592.4597778320312\"]], [[\"39.84159851074219\", \"262.70172119140625\", \"723.0554809570312\", \"568.594482421875\"]], [[\"79.69405364990234\", \"243.9294891357422\", \"726.3303833007812\", \"547.327392578125\"]], [[\"61.52854537963867\", \"329.4207763671875\", \"548.9110107421875\", \"508.7867126464844\"]], [[\"31.69281005859375\", \"325.5604248046875\", \"538.5624389648438\", \"518.9894409179688\"]], [[\"28.702146530151367\", \"348.4562072753906\", \"555.9247436523438\", \"548.6229248046875\"]]]}"878 },879 {880 "id": 5,881 "turn_id": 2,882 "user_query": "Localize the species with the traits of brownish-red body with white speckles, large eyes, tentacles visible, textured skin in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",883 "assistant_response": "<answer>{\"boxes\": [[[\"60.52213668823242\", \"275.92340087890625\", \"724.6060791015625\", \"592.4597778320312\"]], [[\"39.84159851074219\", \"262.70172119140625\", \"723.0554809570312\", \"568.594482421875\"]], [[\"79.69405364990234\", \"243.9294891357422\", \"726.3303833007812\", \"547.327392578125\"]], [[\"61.52854537963867\", \"329.4207763671875\", \"548.9110107421875\", \"508.7867126464844\"]], [[\"31.69281005859375\", \"325.5604248046875\", \"538.5624389648438\", \"518.9894409179688\"]], [[\"28.702146530151367\", \"348.4562072753906\", \"555.9247436523438\", \"548.6229248046875\"]]]}</answer>",884 "visual_input": [885 "frames/frame_000000.jpg",886 "frames/frame_000001.jpg",887 "frames/frame_000002.jpg",888 "frames/frame_000003.jpg",889 "frames/frame_000004.jpg",890 "frames/frame_000005.jpg"891 ]892 }893 ]894 },895 {896 "question": "Localize the species with the traits of dark body with white spots, wide triangular pectoral fins, long tail in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",897 "answer": "{\"boxes\": [[[\"0.655360221862793\", \"612.72607421875\", \"719.0386962890625\", \"1091.6832275390625\"]], [[\"0.6562614440917969\", \"450.0228271484375\", \"538.2191772460938\", \"1064.653076171875\"]], [[\"-1.2963223457336426\", \"439.45556640625\", \"650.3921508789062\", \"1006.4437866210938\"]]]}",898 "explanation": {899 "0": [900 "0.655360221862793",901 "612.72607421875",902 "719.0386962890625",903 "1091.6832275390625"904 ],905 "1": [906 "0.6562614440917969",907 "450.0228271484375",908 "538.2191772460938",909 "1064.653076171875"910 ],911 "2": [912 "-1.2963223457336426",913 "439.45556640625",914 "650.3921508789062",915 "1006.4437866210938"916 ]917 },918 "dimension": "SpatialReasoning",919 "subdimension": "VisualGrounding",920 "question_task": "VideoQuestionAnswering",921 "question_format": "ExactMatchQuestion",922 "video_id": "9LVliZs6jNHF",923 "video_url": "https://www.instagram.com/big.blue.sea/reel/Chzt-BdOX_f",924 "video_scene": "000004_000006",925 "id": 6,926 "original_video_id": "Chzt-BdOX_f",927 "turns": [928 {929 "turn_id": 1,930 "id": 6,931 "assistant_response": "[{\"function\": {\"name\": \"object_grounding\", \"arguments\": {\"prompt\": \"A region of a species with dark body and white spots, wide triangular pectoral fins, and long tail\", \"ground_type\": \"anything\"}}}]",932 "user_query": "Localize the species with the traits of dark body with white spots, wide triangular pectoral fins, long tail in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",933 "visual_input": [934 "frames/frame_000000.jpg",935 "frames/frame_000001.jpg",936 "frames/frame_000002.jpg"937 ],938 "tool_output": "{\"boxes\": [[[\"0.655360221862793\", \"612.72607421875\", \"719.0386962890625\", \"1091.6832275390625\"]], [[\"0.6562614440917969\", \"450.0228271484375\", \"538.2191772460938\", \"1064.653076171875\"]], [[\"-1.2963223457336426\", \"439.45556640625\", \"650.3921508789062\", \"1006.4437866210938\"]]]}"939 },940 {941 "id": 6,942 "turn_id": 2,943 "user_query": "Localize the species with the traits of dark body with white spots, wide triangular pectoral fins, long tail in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",944 "assistant_response": "<answer>{\"boxes\": [[[\"0.655360221862793\", \"612.72607421875\", \"719.0386962890625\", \"1091.6832275390625\"]], [[\"0.6562614440917969\", \"450.0228271484375\", \"538.2191772460938\", \"1064.653076171875\"]], [[\"-1.2963223457336426\", \"439.45556640625\", \"650.3921508789062\", \"1006.4437866210938\"]]]}</answer>",945 "visual_input": [946 "frames/frame_000000.jpg",947 "frames/frame_000001.jpg",948 "frames/frame_000002.jpg"949 ]950 }951 ]952 },953 {954 "question": "Localize the species with the traits of yellowish-orange color, five arms, attached to rocks in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",955 "answer": "{\"boxes\": [[[\"114.92340850830078\", \"490.6473388671875\", \"640.6270141601562\", \"969.642578125\"]], [[\"112.12902069091797\", \"503.02813720703125\", \"642.7485961914062\", \"988.25927734375\"]], [[\"107.78073120117188\", \"510.8506774902344\", \"639.9957275390625\", \"993.0845947265625\"]], [[\"110.09690856933594\", \"499.17852783203125\", \"639.3856811523438\", \"977.4501953125\"]]]}",956 "explanation": {957 "0": [958 "114.92340850830078",959 "490.6473388671875",960 "640.6270141601562",961 "969.642578125"962 ],963 "1": [964 "112.12902069091797",965 "503.02813720703125",966 "642.7485961914062",967 "988.25927734375"968 ],969 "2": [970 "107.78073120117188",971 "510.8506774902344",972 "639.9957275390625",973 "993.0845947265625"974 ],975 "3": [976 "110.09690856933594",977 "499.17852783203125",978 "639.3856811523438",979 "977.4501953125"980 ]981 },982 "dimension": "SpatialReasoning",983 "subdimension": "VisualGrounding",984 "question_task": "VideoQuestionAnswering",985 "question_format": "ExactMatchQuestion",986 "video_id": "fEfr8fdSdNJM",987 "video_url": "https://www.instagram.com/big.blue.sea/reel/Cr8JSGdv4Dx",988 "video_scene": "000007_000010",989 "id": 7,990 "original_video_id": "Cr8JSGdv4Dx",991 "turns": [992 {993 "turn_id": 1,994 "id": 7,995 "assistant_response": "[{\"function\": {\"name\": \"object_grounding\", \"arguments\": {\"prompt\": \"A region of a species with yellowish-orange color and five arms attached to rocks\", \"ground_type\": \"anything\"}}}]",996 "user_query": "Localize the species with the traits of yellowish-orange color, five arms, attached to rocks in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",997 "visual_input": [998 "frames/frame_000000.jpg",999 "frames/frame_000001.jpg",1000 "frames/frame_000002.jpg",1001 "frames/frame_000003.jpg"1002 ],1003 "tool_output": "{\"boxes\": [[[\"114.92340850830078\", \"490.6473388671875\", \"640.6270141601562\", \"969.642578125\"]], [[\"112.12902069091797\", \"503.02813720703125\", \"642.7485961914062\", \"988.25927734375\"]], [[\"107.78073120117188\", \"510.8506774902344\", \"639.9957275390625\", \"993.0845947265625\"]], [[\"110.09690856933594\", \"499.17852783203125\", \"639.3856811523438\", \"977.4501953125\"]]]}"1004 },1005 {1006 "id": 7,1007 "turn_id": 2,1008 "user_query": "Localize the species with the traits of yellowish-orange color, five arms, attached to rocks in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",1009 "assistant_response": "<answer>{\"boxes\": [[[\"114.92340850830078\", \"490.6473388671875\", \"640.6270141601562\", \"969.642578125\"]], [[\"112.12902069091797\", \"503.02813720703125\", \"642.7485961914062\", \"988.25927734375\"]], [[\"107.78073120117188\", \"510.8506774902344\", \"639.9957275390625\", \"993.0845947265625\"]], [[\"110.09690856933594\", \"499.17852783203125\", \"639.3856811523438\", \"977.4501953125\"]]]}</answer>",1010 "visual_input": [1011 "frames/frame_000000.jpg",1012 "frames/frame_000001.jpg",1013 "frames/frame_000002.jpg",1014 "frames/frame_000003.jpg"1015 ]1016 }1017 ]1018 },1019 {1020 "question": "Localize the species with the traits of mottled brown to reddish-brown body with darker bars, distinct banded pectoral fins, short tentacle over the eye, leaf-like appendages on head and lateral line, black spots edged with red on filaments from dorsal spines in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",1021 "answer": "{\"boxes\": [[[\"342.15716552734375\", \"705.4196166992188\", \"650.2496948242188\", \"956.44091796875\"]], [[\"276.3952331542969\", \"737.1573486328125\", \"604.3368530273438\", \"980.2388916015625\"]], [[\"264.87762451171875\", \"722.434326171875\", \"601.705078125\", \"950.1419677734375\"]], [[\"255.55563354492188\", \"721.4099731445312\", \"612.2300415039062\", \"919.64013671875\"]]]}",1022 "explanation": {1023 "0": [1024 "342.15716552734375",1025 "705.4196166992188",1026 "650.2496948242188",1027 "956.44091796875"1028 ],1029 "1": [1030 "276.3952331542969",1031 "737.1573486328125",1032 "604.3368530273438",1033 "980.2388916015625"1034 ],1035 "2": [1036 "264.87762451171875",1037 "722.434326171875",1038 "601.705078125",1039 "950.1419677734375"1040 ],1041 "3": [1042 "255.55563354492188",1043 "721.4099731445312",1044 "612.2300415039062",1045 "919.64013671875"1046 ]1047 },1048 "dimension": "SpatialReasoning",1049 "subdimension": "VisualGrounding",1050 "question_task": "VideoQuestionAnswering",1051 "question_format": "ExactMatchQuestion",1052 "video_id": "wUjOI72J7C9r",1053 "video_url": "https://www.instagram.com/big.blue.sea/reel/Cx5dkTnNz0-",1054 "video_scene": "000001_000004",1055 "id": 8,1056 "original_video_id": "Cx5dkTnNz0-",1057 "turns": [1058 {1059 "turn_id": 1,1060 "id": 8,1061 "assistant_response": "[{\"function\": {\"name\": \"object_grounding\", \"arguments\": {\"prompt\": \"A region of a fish with a mottled brown to reddish-brown body, darker bars, distinct banded pectoral fins, a short tentacle over the eye, leaf-like appendages on the head and lateral line, and black spots edged with red on filaments from dorsal spines\", \"ground_type\": \"anything\"}}}]",1062 "user_query": "Localize the species with the traits of mottled brown to reddish-brown body with darker bars, distinct banded pectoral fins, short tentacle over the eye, leaf-like appendages on head and lateral line, black spots edged with red on filaments from dorsal spines in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",1063 "visual_input": [1064 "frames/frame_000000.jpg",1065 "frames/frame_000001.jpg",1066 "frames/frame_000002.jpg",1067 "frames/frame_000003.jpg"1068 ],1069 "tool_output": "{\"boxes\": [[[\"342.15716552734375\", \"705.4196166992188\", \"650.2496948242188\", \"956.44091796875\"]], [[\"276.3952331542969\", \"737.1573486328125\", \"604.3368530273438\", \"980.2388916015625\"]], [[\"264.87762451171875\", \"722.434326171875\", \"601.705078125\", \"950.1419677734375\"]], [[\"255.55563354492188\", \"721.4099731445312\", \"612.2300415039062\", \"919.64013671875\"]]]}"1070 },1071 {1072 "id": 8,1073 "turn_id": 2,1074 "user_query": "Localize the species with the traits of mottled brown to reddish-brown body with darker bars, distinct banded pectoral fins, short tentacle over the eye, leaf-like appendages on head and lateral line, black spots edged with red on filaments from dorsal spines in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",1075 "assistant_response": "<answer>{\"boxes\": [[[\"342.15716552734375\", \"705.4196166992188\", \"650.2496948242188\", \"956.44091796875\"]], [[\"276.3952331542969\", \"737.1573486328125\", \"604.3368530273438\", \"980.2388916015625\"]], [[\"264.87762451171875\", \"722.434326171875\", \"601.705078125\", \"950.1419677734375\"]], [[\"255.55563354492188\", \"721.4099731445312\", \"612.2300415039062\", \"919.64013671875\"]]]}</answer>",1076 "visual_input": [1077 "frames/frame_000000.jpg",1078 "frames/frame_000001.jpg",1079 "frames/frame_000002.jpg",1080 "frames/frame_000003.jpg"1081 ]1082 }1083 ]1084 },1085 {1086 "question": "Localize the species with the traits of wearing full diving gear including mask, tank, and fins in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",1087 "answer": "{\"boxes\": [[[\"455.7377014160156\", \"683.618408203125\", \"723.5035400390625\", \"1226.0224609375\"]], [[\"451.75823974609375\", \"658.6019287109375\", \"725.711181640625\", \"1180.5655517578125\"]], [[\"453.9407043457031\", \"598.3863525390625\", \"723.80615234375\", \"1082.250732421875\"]], [[\"414.9433898925781\", \"532.9330444335938\", \"686.07763671875\", \"1003.6968994140625\"]]]}",1088 "explanation": {1089 "0": [1090 "455.7377014160156",1091 "683.618408203125",1092 "723.5035400390625",1093 "1226.0224609375"1094 ],1095 "1": [1096 "451.75823974609375",1097 "658.6019287109375",1098 "725.711181640625",1099 "1180.5655517578125"1100 ],1101 "2": [1102 "453.9407043457031",1103 "598.3863525390625",1104 "723.80615234375",1105 "1082.250732421875"1106 ],1107 "3": [1108 "414.9433898925781",1109 "532.9330444335938",1110 "686.07763671875",1111 "1003.6968994140625"1112 ]1113 },1114 "dimension": "SpatialReasoning",1115 "subdimension": "VisualGrounding",1116 "question_task": "VideoQuestionAnswering",1117 "question_format": "ExactMatchQuestion",1118 "video_id": "Kx4OxIWWxhfO",1119 "video_url": "https://www.instagram.com/tomparkfilms/reel/CxY-5MTSadG",1120 "video_scene": "000007_000010",1121 "id": 9,1122 "original_video_id": "CxY-5MTSadG",1123 "turns": [1124 {1125 "turn_id": 1,1126 "id": 9,1127 "assistant_response": "[{\"function\": {\"name\": \"object_grounding\", \"arguments\": {\"prompt\": \"A region of a species wearing full diving gear including mask, tank, and fins\", \"ground_type\": \"anything\"}}}]",1128 "user_query": "Localize the species with the traits of wearing full diving gear including mask, tank, and fins in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",1129 "visual_input": [1130 "frames/frame_000000.jpg",1131 "frames/frame_000001.jpg",1132 "frames/frame_000002.jpg",1133 "frames/frame_000003.jpg"1134 ],1135 "tool_output": "{\"boxes\": [[[\"455.7377014160156\", \"683.618408203125\", \"723.5035400390625\", \"1226.0224609375\"]], [[\"451.75823974609375\", \"658.6019287109375\", \"725.711181640625\", \"1180.5655517578125\"]], [[\"453.9407043457031\", \"598.3863525390625\", \"723.80615234375\", \"1082.250732421875\"]], [[\"414.9433898925781\", \"532.9330444335938\", \"686.07763671875\", \"1003.6968994140625\"]]]}"1136 },1137 {1138 "id": 9,1139 "turn_id": 2,1140 "user_query": "Localize the species with the traits of wearing full diving gear including mask, tank, and fins in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",1141 "assistant_response": "<answer>{\"boxes\": [[[\"455.7377014160156\", \"683.618408203125\", \"723.5035400390625\", \"1226.0224609375\"]], [[\"451.75823974609375\", \"658.6019287109375\", \"725.711181640625\", \"1180.5655517578125\"]], [[\"453.9407043457031\", \"598.3863525390625\", \"723.80615234375\", \"1082.250732421875\"]], [[\"414.9433898925781\", \"532.9330444335938\", \"686.07763671875\", \"1003.6968994140625\"]]]}</answer>",1142 "visual_input": [1143 "frames/frame_000000.jpg",1144 "frames/frame_000001.jpg",1145 "frames/frame_000002.jpg",1146 "frames/frame_000003.jpg"1147 ]1148 }1149 ]1150 },1151 {1152 "question": "Localize the species with the traits of bright blue color, large body, distinct dorsal fin with a trigger-like mechanism, yellow eyes, pattern of orange lines on the face in later frames in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",1153 "answer": "{\"boxes\": [[[\"-1.0000133514404297\", \"580.5753173828125\", \"717.4625244140625\", \"1280.71630859375\"]], [[\"0.3637075424194336\", \"565.188232421875\", \"718.2703857421875\", \"1279.794189453125\"]], [[\"-0.5918025970458984\", \"563.407470703125\", \"717.9193115234375\", \"1278.772705078125\"]]]}",1154 "explanation": {1155 "0": [1156 "-1.0000133514404297",1157 "580.5753173828125",1158 "717.4625244140625",1159 "1280.71630859375"1160 ],1161 "1": [1162 "0.3637075424194336",1163 "565.188232421875",1164 "718.2703857421875",1165 "1279.794189453125"1166 ],1167 "2": [1168 "-0.5918025970458984",1169 "563.407470703125",1170 "717.9193115234375",1171 "1278.772705078125"1172 ]1173 },1174 "dimension": "SpatialReasoning",1175 "subdimension": "VisualGrounding",1176 "question_task": "VideoQuestionAnswering",1177 "question_format": "ExactMatchQuestion",1178 "video_id": "iRx4Gh4FcjDR",1179 "video_url": "https://www.instagram.com/big.blue.sea/reel/DFK03eRN0O5",1180 "video_scene": "000007_000009",1181 "id": 10,1182 "original_video_id": "DFK03eRN0O5",1183 "turns": [1184 {1185 "turn_id": 1,1186 "id": 10,1187 "assistant_response": "[{\"function\": {\"name\": \"object_grounding\", \"arguments\": {\"prompt\": \"A region of a species with bright blue color, large body, distinct dorsal fin with a trigger-like mechanism, yellow eyes, and orange lines on the face\", \"ground_type\": \"anything\"}}}]",1188 "user_query": "Localize the species with the traits of bright blue color, large body, distinct dorsal fin with a trigger-like mechanism, yellow eyes, pattern of orange lines on the face in later frames in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",1189 "visual_input": [1190 "frames/frame_000000.jpg",1191 "frames/frame_000001.jpg",1192 "frames/frame_000002.jpg"1193 ],1194 "tool_output": "{\"boxes\": [[[\"-1.0000133514404297\", \"580.5753173828125\", \"717.4625244140625\", \"1280.71630859375\"]], [[\"0.3637075424194336\", \"565.188232421875\", \"718.2703857421875\", \"1279.794189453125\"]], [[\"-0.5918025970458984\", \"563.407470703125\", \"717.9193115234375\", \"1278.772705078125\"]]]}"1195 },1196 {1197 "id": 10,1198 "turn_id": 2,1199 "user_query": "Localize the species with the traits of bright blue color, large body, distinct dorsal fin with a trigger-like mechanism, yellow eyes, pattern of orange lines on the face in later frames in the video. Provide your answer as a dictionary where each key is a timestamp in seconds (as a string), and each value is the corresponding bounding box in the format [x_min, y_min, x_max, y_max]. For example: {'0': [100, 200, 300, 400]}.",1200 "assistant_response": "<answer>{\"boxes\": [[[\"-1.0000133514404297\", \"580.5753173828125\", \"717.4625244140625\", \"1280.71630859375\"]], [[\"0.3637075424194336\", \"565.188232421875\", \"718.2703857421875\", \"1279.794189453125\"]], [[\"-0.5918025970458984\", \"563.407470703125\", \"717.9193115234375\", \"1278.772705078125\"]]]}</answer>",