Greatriver/SimBarca

1INFO: Pandarallel will run on 8 workers.2INFO: Pandarallel will use Memory file system to transfer data between the main process and workers.3 4 Selecting 5.0% vehicles from 179806 vehicles5Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/00_traj_1261.5.json.gz6Loading a file takes 41.81s7Creating data frames takes 31.49s8Filling data types and values takes 6.14s9 10 Processing a file takes 17.57s11Concatenating time series takes 0.44s12Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/01_traj_1831.5.json.gz13Loading a file takes 52.42s14Creating data frames takes 22.85s15Filling data types and values takes 5.71s16 17 Processing a file takes 16.69s18Concatenating time series takes 0.26s19Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/02_traj_2338.5.json.gz20Loading a file takes 53.68s21Creating data frames takes 23.45s22Filling data types and values takes 5.75s23 24 Processing a file takes 16.50s25Concatenating time series takes 0.21s26Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/03_traj_2810.5.json.gz27Loading a file takes 53.47s28Creating data frames takes 19.86s29Filling data types and values takes 5.87s30 31 Processing a file takes 16.53s32Concatenating time series takes 0.36s33Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/04_traj_3249.0.json.gz34Loading a file takes 54.26s35Creating data frames takes 19.53s36Filling data types and values takes 5.69s37 38 Processing a file takes 15.97s39Concatenating time series takes 0.30s40Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/05_traj_3662.5.json.gz41Loading a file takes 51.99s42Creating data frames takes 20.46s43Filling data types and values takes 5.77s44 45 Processing a file takes 16.13s46Concatenating time series takes 0.38s47Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/06_traj_4058.0.json.gz48Loading a file takes 52.72s49Creating data frames takes 20.38s50Filling data types and values takes 5.86s51 52 Processing a file takes 16.16s53Concatenating time series takes 0.37s54Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/07_traj_4435.0.json.gz55Loading a file takes 55.78s56Creating data frames takes 20.46s57Filling data types and values takes 6.03s58 59 Processing a file takes 16.50s60Concatenating time series takes 0.36s61Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/08_traj_4792.5.json.gz62Loading a file takes 54.94s63Creating data frames takes 21.27s64Filling data types and values takes 6.03s65 66 Processing a file takes 17.10s67Concatenating time series takes 0.31s68Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/09_traj_5127.0.json.gz69Loading a file takes 57.55s70Creating data frames takes 22.99s71Filling data types and values takes 5.99s72 73 Processing a file takes 16.74s74Concatenating time series takes 0.43s75Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/10_traj_5435.5.json.gz76Loading a file takes 57.70s77Creating data frames takes 24.69s78Filling data types and values takes 5.89s79 80 Processing a file takes 17.36s81Concatenating time series takes 0.45s82Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/11_traj_5726.5.json.gz83Loading a file takes 57.51s84Creating data frames takes 25.16s85Filling data types and values takes 5.99s86 87 Processing a file takes 16.94s88Concatenating time series takes 0.26s89Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/12_traj_6007.5.json.gz90Loading a file takes 54.81s91Creating data frames takes 31.67s92Filling data types and values takes 5.99s93 94 Processing a file takes 16.66s95Concatenating time series takes 0.45s96Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/13_traj_6266.5.json.gz97Loading a file takes 57.71s98Creating data frames takes 25.38s99Filling data types and values takes 6.00s100 101 Processing a file takes 17.00s102Concatenating time series takes 0.39s103Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/14_traj_6505.5.json.gz104Loading a file takes 55.16s105Creating data frames takes 24.81s106Filling data types and values takes 5.31s107 108 Processing a file takes 16.53s109Concatenating time series takes 0.26s110Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/15_traj_6731.5.json.gz111Loading a file takes 46.64s112Creating data frames takes 22.58s113Filling data types and values takes 5.40s114 115 Processing a file takes 16.15s116Concatenating time series takes 0.29s117Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/16_traj_6946.5.json.gz118Loading a file takes 50.90s119Creating data frames takes 22.42s120Filling data types and values takes 5.27s121 122 Processing a file takes 15.89s123Concatenating time series takes 0.33s124Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/17_traj_7150.0.json.gz125Loading a file takes 50.02s126Creating data frames takes 41.68s127Filling data types and values takes 6.41s128 129 Processing a file takes 18.25s130Concatenating time series takes 0.38s131Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/18_traj_7350.5.json.gz132Loading a file takes 61.27s133Creating data frames takes 29.65s134Filling data types and values takes 5.88s135 136 Processing a file takes 17.79s137Concatenating time series takes 0.31s138Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/19_traj_7563.0.json.gz139Loading a file takes 62.65s140Creating data frames takes 25.33s141Filling data types and values takes 5.99s142 143 Processing a file takes 17.09s144Concatenating time series takes 0.32s145Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/20_traj_7791.5.json.gz146Loading a file takes 62.55s147Creating data frames takes 24.63s148Filling data types and values takes 6.01s149 150 Processing a file takes 17.17s151Concatenating time series takes 0.31s152Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/21_traj_8041.0.json.gz153Loading a file takes 57.65s154Creating data frames takes 27.66s155Filling data types and values takes 6.03s156 157 Processing a file takes 17.11s158Concatenating time series takes 0.43s159Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/22_traj_8318.0.json.gz160Loading a file takes 61.52s161Creating data frames takes 24.89s162Filling data types and values takes 6.04s163 164 Processing a file takes 17.64s165Concatenating time series takes 0.33s166Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/23_traj_8625.0.json.gz167Loading a file takes 62.02s168Creating data frames takes 30.64s169Filling data types and values takes 6.18s170 171 Processing a file takes 17.09s172Concatenating time series takes 0.45s173Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/24_traj_8969.5.json.gz174Loading a file takes 57.03s175Creating data frames takes 25.71s176Filling data types and values takes 6.44s177 178 Processing a file takes 17.16s179Concatenating time series takes 0.39s180Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/25_traj_9367.5.json.gz181Loading a file takes 62.75s182Creating data frames takes 29.57s183Filling data types and values takes 6.48s184 185 Processing a file takes 15.55s186Concatenating time series takes 0.41s187Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/26_traj_9822.5.json.gz188Loading a file takes 58.51s189Creating data frames takes 28.91s190Filling data types and values takes 5.46s191 192 Processing a file takes 16.01s193Concatenating time series takes 0.43s194Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/27_traj_10337.0.json.gz195Loading a file takes 58.65s196Creating data frames takes 67.66s197Filling data types and values takes 6.90s198 199 Processing a file takes 14.95s200Concatenating time series takes 0.41s201Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/28_traj_10915.0.json.gz202Loading a file takes 59.86s203Creating data frames takes 62.85s204Filling data types and values takes 6.74s205 206 Processing a file takes 15.62s207Concatenating time series takes 0.33s208Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/29_traj_11560.5.json.gz209Loading a file takes 47.86s210Creating data frames takes 25.06s211Filling data types and values takes 6.07s212 213 Processing a file takes 30.87s214Concatenating time series takes 0.21s215Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/30_traj_12271.0.json.gz216Loading a file takes 60.53s217Creating data frames takes 28.13s218Filling data types and values takes 6.30s219 220 Processing a file takes 14.50s221Concatenating time series takes 0.21s222Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/31_traj_13031.5.json.gz223Loading a file takes 57.29s224Creating data frames takes 28.01s225Filling data types and values takes 6.66s226 227 Processing a file takes 14.96s228Concatenating time series takes 0.27s229Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/32_traj_13828.0.json.gz230Loading a file takes 59.30s231Creating data frames takes 28.04s232Filling data types and values takes 6.69s233 234 Processing a file takes 13.37s235Concatenating time series takes 0.23s236Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/33_traj_14653.0.json.gz237Loading a file takes 64.12s238Creating data frames takes 27.79s239Filling data types and values takes 6.52s240 241 Processing a file takes 13.36s242Concatenating time series takes 0.20s243Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/34_traj_15498.0.json.gz244Loading a file takes 57.30s245Creating data frames takes 28.39s246Filling data types and values takes 6.38s247 248 Processing a file takes 12.83s249Concatenating time series takes 0.19s250Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/35_traj_16348.5.json.gz251Loading a file takes 61.20s252Creating data frames takes 26.29s253Filling data types and values takes 6.52s254 255 Processing a file takes 12.22s256Concatenating time series takes 0.14s257Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/36_traj_17203.0.json.gz258Loading a file takes 60.78s259Creating data frames takes 25.43s260Filling data types and values takes 6.28s261 262 Processing a file takes 12.26s263Concatenating time series takes 0.14s264Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/37_traj_18063.0.json.gz265Loading a file takes 56.08s266Creating data frames takes 24.35s267Filling data types and values takes 6.08s268 269 Processing a file takes 11.86s270Concatenating time series takes 0.14s271Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/38_traj_18927.0.json.gz272Loading a file takes 62.13s273Creating data frames takes 25.60s274Filling data types and values takes 6.39s275 276 Processing a file takes 12.12s277Concatenating time series takes 0.13s278Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/39_traj_19792.0.json.gz279Loading a file takes 56.32s280Creating data frames takes 25.47s281Filling data types and values takes 6.04s282 283 Processing a file takes 11.60s284Concatenating time series takes 0.20s285Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/40_traj_20659.0.json.gz286Loading a file takes 60.58s287Creating data frames takes 25.60s288Filling data types and values takes 6.30s289 290 Processing a file takes 12.07s291Concatenating time series takes 0.15s292Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/41_traj_21529.5.json.gz293Loading a file takes 53.34s294Creating data frames takes 24.32s295Filling data types and values takes 5.97s296 297 Processing a file takes 11.97s298Concatenating time series takes 0.17s299Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/42_traj_22402.5.json.gz300Loading a file takes 58.45s301Creating data frames takes 24.06s302Filling data types and values takes 6.41s303 304 Processing a file takes 11.72s305Concatenating time series takes 0.15s306Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/43_traj_23276.5.json.gz307Loading a file takes 52.89s308Creating data frames takes 22.02s309Filling data types and values takes 5.99s310 311 Processing a file takes 11.28s312Concatenating time series takes 0.15s313Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/trajectory/44_traj_end.json.gz314Loading a file takes 7.08s315Creating data frames takes 2.99s316Filling data types and values takes 0.48s317 318 Processing a file takes 5.62s319Concatenating time series takes 0.04s320Saving all the section time series takes 144.78s321Saving all the junction time series takes 22.19s322Saving network entrance and exit takes 0.14s323Aggregating the raw statistics to different intervals...324
  0%|          | 0/1525 [00:00<?, ?it/s]
  0%|          | 1/1525 [00:00<23:51,  1.06it/s]
  0%|          | 5/1525 [00:01<04:16,  5.93it/s]
  1%|          | 19/1525 [00:01<00:57, 26.33it/s]
  3%|▎         | 40/1525 [00:01<00:25, 58.05it/s]
  3%|▎         | 52/1525 [00:01<00:22, 65.09it/s]
  5%|▍         | 69/1525 [00:01<00:17, 82.62it/s]
  6%|▋         | 97/1525 [00:01<00:11, 124.12it/s]
  7%|▋         | 114/1525 [00:01<00:11, 123.44it/s]
  9%|▉         | 134/1525 [00:01<00:10, 138.61it/s]
 10%|▉         | 151/1525 [00:02<00:09, 140.31it/s]
 11%|█         | 167/1525 [00:02<00:10, 132.71it/s]
 12%|█▏        | 184/1525 [00:02<00:09, 140.95it/s]
 13%|█▎        | 200/1525 [00:02<00:09, 140.98it/s]
 14%|█▍        | 219/1525 [00:02<00:08, 151.77it/s]
 15%|█▌        | 235/1525 [00:02<00:09, 141.90it/s]
 16%|█▋        | 250/1525 [00:02<00:09, 136.23it/s]
 17%|█▋        | 264/1525 [00:02<00:11, 105.72it/s]
 18%|█▊        | 278/1525 [00:03<00:11, 111.43it/s]
 19%|█▉        | 293/1525 [00:03<00:10, 118.26it/s]
 20%|██        | 312/1525 [00:04<00:45, 26.62it/s] 
 22%|██▏       | 333/1525 [00:05<00:31, 38.38it/s]
 23%|██▎       | 353/1525 [00:05<00:22, 51.02it/s]
 25%|██▌       | 387/1525 [00:05<00:13, 81.55it/s]
 27%|██▋       | 407/1525 [00:05<00:11, 94.97it/s]
 28%|██▊       | 426/1525 [00:05<00:10, 104.20it/s]
 29%|██▉       | 444/1525 [00:05<00:09, 110.67it/s]
 30%|███       | 461/1525 [00:05<00:09, 116.03it/s]
 31%|███▏      | 477/1525 [00:05<00:08, 123.86it/s]
 32%|███▏      | 493/1525 [00:06<00:09, 109.43it/s]
 33%|███▎      | 507/1525 [00:06<00:08, 114.32it/s]
 34%|███▍      | 521/1525 [00:06<00:08, 117.91it/s]
 35%|███▌      | 535/1525 [00:06<00:08, 119.80it/s]
 36%|███▌      | 548/1525 [00:06<00:09, 99.64it/s] 
 37%|███▋      | 560/1525 [00:06<00:13, 73.26it/s]
 37%|███▋      | 570/1525 [00:07<00:14, 67.30it/s]
 39%|███▉      | 591/1525 [00:07<00:10, 92.76it/s]
 40%|███▉      | 604/1525 [00:07<00:09, 93.08it/s]
 40%|████      | 615/1525 [00:07<00:12, 72.20it/s]
 42%|████▏     | 646/1525 [00:07<00:07, 115.85it/s]
 44%|████▎     | 664/1525 [00:07<00:06, 126.92it/s]
 45%|████▍     | 682/1525 [00:07<00:06, 135.69it/s]
 46%|████▌     | 699/1525 [00:07<00:05, 139.61it/s]
 47%|████▋     | 717/1525 [00:08<00:05, 147.77it/s]
 48%|████▊     | 734/1525 [00:08<00:05, 150.98it/s]
 49%|████▉     | 751/1525 [00:08<00:05, 142.88it/s]
 50%|█████     | 766/1525 [00:08<00:05, 136.91it/s]
 51%|█████     | 781/1525 [00:08<00:06, 122.55it/s]
 52%|█████▏    | 796/1525 [00:08<00:05, 127.56it/s]
 53%|█████▎    | 811/1525 [00:08<00:05, 132.24it/s]
 54%|█████▍    | 825/1525 [00:08<00:05, 117.45it/s]
 55%|█████▍    | 838/1525 [00:09<00:06, 114.02it/s]
 56%|█████▌    | 852/1525 [00:09<00:06, 111.98it/s]
 57%|█████▋    | 866/1525 [00:09<00:06, 108.43it/s]
 58%|█████▊    | 878/1525 [00:09<00:06, 106.61it/s]
 58%|█████▊    | 889/1525 [00:09<00:06, 98.38it/s] 
 59%|█████▉    | 902/1525 [00:09<00:05, 105.60it/s]
 60%|██████    | 917/1525 [00:09<00:05, 115.22it/s]
 61%|██████    | 932/1525 [00:09<00:04, 121.13it/s]
 62%|██████▏   | 945/1525 [00:10<00:05, 112.93it/s]
 64%|██████▍   | 974/1525 [00:10<00:03, 158.66it/s]
 65%|██████▍   | 991/1525 [00:10<00:03, 148.27it/s]
 66%|██████▌   | 1007/1525 [00:10<00:04, 128.54it/s]
 67%|██████▋   | 1021/1525 [00:10<00:04, 105.97it/s]
 68%|██████▊   | 1033/1525 [00:10<00:05, 86.76it/s] 
 69%|██████▊   | 1045/1525 [00:11<00:05, 91.95it/s]
 70%|██████▉   | 1060/1525 [00:11<00:04, 104.37it/s]
 70%|███████   | 1073/1525 [00:11<00:04, 107.51it/s]
 71%|███████   | 1085/1525 [00:11<00:04, 104.24it/s]
 72%|███████▏  | 1099/1525 [00:11<00:03, 112.45it/s]
 73%|███████▎  | 1113/1525 [00:11<00:03, 118.94it/s]
 74%|███████▍  | 1126/1525 [00:11<00:04, 97.30it/s] 
 75%|███████▌  | 1146/1525 [00:11<00:03, 121.54it/s]
 76%|███████▌  | 1162/1525 [00:11<00:03, 116.20it/s]
 77%|███████▋  | 1177/1525 [00:12<00:02, 121.37it/s]
 78%|███████▊  | 1191/1525 [00:12<00:02, 125.34it/s]
 79%|███████▉  | 1207/1525 [00:12<00:02, 123.41it/s]
 80%|████████  | 1223/1525 [00:12<00:02, 126.85it/s]
 82%|████████▏ | 1244/1525 [00:12<00:02, 132.43it/s]
 83%|████████▎ | 1260/1525 [00:12<00:01, 139.18it/s]
 84%|████████▍ | 1283/1525 [00:12<00:01, 155.67it/s]
 85%|████████▌ | 1303/1525 [00:12<00:01, 165.92it/s]
 87%|████████▋ | 1320/1525 [00:13<00:01, 153.13it/s]
 88%|████████▊ | 1336/1525 [00:13<00:01, 146.15it/s]
 89%|████████▉ | 1355/1525 [00:13<00:01, 155.87it/s]
 90%|████████▉ | 1371/1525 [00:15<00:07, 21.62it/s] 
 91%|█████████ | 1383/1525 [00:15<00:05, 25.73it/s]
 91%|█████████▏| 1394/1525 [00:15<00:04, 30.96it/s]
 93%|█████████▎| 1417/1525 [00:16<00:02, 47.62it/s]
 94%|█████████▍| 1438/1525 [00:16<00:01, 64.09it/s]
 95%|█████████▌| 1454/1525 [00:16<00:01, 70.65it/s]
 96%|█████████▋| 1470/1525 [00:16<00:00, 82.10it/s]
 97%|█████████▋| 1486/1525 [00:16<00:00, 88.45it/s]
 99%|█████████▊| 1504/1525 [00:16<00:00, 104.74it/s]
100%|██████████| 1525/1525 [00:16<00:00, 90.67it/s] 325Constructing time series for each modality ...326Saving aggregated time series to /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_091/timeseries/agg_timeseries.pkl327Extracting samples from time series ...328number of samples: 20329Packing the samples into numpy arrays ...330