Greatriver/SimBarca

1INFO: Pandarallel will run on 8 workers.2INFO: Pandarallel will use Memory file system to transfer data between the main process and workers.3 4 Selecting 5.0% vehicles from 178925 vehicles5Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/00_traj_1251.5.json.gz6Loading a file takes 49.48s7Creating data frames takes 18.31s8Filling data types and values takes 5.61s9 10 Processing a file takes 17.71s11Concatenating time series takes 0.51s12Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/01_traj_1827.0.json.gz13Loading a file takes 58.17s14Creating data frames takes 20.89s15Filling data types and values takes 6.03s16 17 Processing a file takes 17.56s18Concatenating time series takes 0.27s19Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/02_traj_2327.0.json.gz20Loading a file takes 57.90s21Creating data frames takes 20.35s22Filling data types and values takes 5.90s23 24 Processing a file takes 17.52s25Concatenating time series takes 0.33s26Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/03_traj_2799.0.json.gz27Loading a file takes 54.33s28Creating data frames takes 19.90s29Filling data types and values takes 5.81s30 31 Processing a file takes 16.77s32Concatenating time series takes 0.34s33Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/04_traj_3251.0.json.gz34Loading a file takes 57.92s35Creating data frames takes 20.49s36Filling data types and values takes 6.06s37 38 Processing a file takes 17.20s39Concatenating time series takes 0.29s40Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/05_traj_3677.5.json.gz41Loading a file takes 52.28s42Creating data frames takes 19.97s43Filling data types and values takes 5.73s44 45 Processing a file takes 16.44s46Concatenating time series takes 0.44s47Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/06_traj_4084.5.json.gz48Loading a file takes 54.56s49Creating data frames takes 18.10s50Filling data types and values takes 5.39s51 52 Processing a file takes 15.81s53Concatenating time series takes 0.37s54Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/07_traj_4472.0.json.gz55Loading a file takes 52.38s56Creating data frames takes 18.91s57Filling data types and values takes 5.72s58 59 Processing a file takes 15.74s60Concatenating time series takes 0.40s61Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/08_traj_4828.0.json.gz62Loading a file takes 51.92s63Creating data frames takes 21.11s64Filling data types and values takes 5.91s65 66 Processing a file takes 18.03s67Concatenating time series takes 0.43s68Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/09_traj_5157.5.json.gz69Loading a file takes 57.36s70Creating data frames takes 26.42s71Filling data types and values takes 6.50s72 73 Processing a file takes 17.84s74Concatenating time series takes 0.40s75Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/10_traj_5465.0.json.gz76Loading a file takes 57.28s77Creating data frames takes 25.11s78Filling data types and values takes 6.72s79 80 Processing a file takes 17.58s81Concatenating time series takes 0.54s82Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/11_traj_5756.0.json.gz83Loading a file takes 58.45s84Creating data frames takes 24.44s85Filling data types and values takes 7.04s86 87 Processing a file takes 17.93s88Concatenating time series takes 0.33s89Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/12_traj_6023.0.json.gz90Loading a file takes 59.07s91Creating data frames takes 31.72s92Filling data types and values takes 5.86s93 94 Processing a file takes 17.22s95Concatenating time series takes 0.43s96Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/13_traj_6270.0.json.gz97Loading a file takes 55.49s98Creating data frames takes 24.07s99Filling data types and values takes 7.20s100 101 Processing a file takes 17.53s102Concatenating time series takes 0.42s103Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/14_traj_6503.0.json.gz104Loading a file takes 51.45s105Creating data frames takes 23.33s106Filling data types and values takes 6.83s107 108 Processing a file takes 18.02s109Concatenating time series takes 0.31s110Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/15_traj_6721.5.json.gz111Loading a file takes 56.51s112Creating data frames takes 21.48s113Filling data types and values takes 5.96s114 115 Processing a file takes 16.56s116Concatenating time series takes 0.24s117Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/16_traj_6927.5.json.gz118Loading a file takes 53.48s119Creating data frames takes 21.75s120Filling data types and values takes 5.83s121 122 Processing a file takes 17.16s123Concatenating time series takes 0.42s124Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/17_traj_7125.0.json.gz125Loading a file takes 57.06s126Creating data frames takes 23.09s127Filling data types and values takes 6.34s128 129 Processing a file takes 17.32s130Concatenating time series takes 0.41s131Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/18_traj_7317.5.json.gz132Loading a file takes 58.33s133Creating data frames takes 24.57s134Filling data types and values takes 7.05s135 136 Processing a file takes 18.14s137Concatenating time series takes 0.29s138Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/19_traj_7521.0.json.gz139Loading a file takes 58.01s140Creating data frames takes 25.94s141Filling data types and values takes 7.45s142 143 Processing a file takes 17.77s144Concatenating time series takes 0.35s145Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/20_traj_7738.5.json.gz146Loading a file takes 61.91s147Creating data frames takes 22.11s148Filling data types and values takes 5.79s149 150 Processing a file takes 15.99s151Concatenating time series takes 0.45s152Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/21_traj_7973.5.json.gz153Loading a file takes 54.62s154Creating data frames takes 25.35s155Filling data types and values takes 7.52s156 157 Processing a file takes 17.98s158Concatenating time series takes 0.46s159Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/22_traj_8225.5.json.gz160Loading a file takes 50.33s161Creating data frames takes 23.51s162Filling data types and values takes 7.95s163 164 Processing a file takes 16.12s165Concatenating time series takes 0.54s166Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/23_traj_8491.5.json.gz167Loading a file takes 54.41s168Creating data frames takes 25.68s169Filling data types and values takes 7.65s170 171 Processing a file takes 18.37s172Concatenating time series takes 0.31s173Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/24_traj_8776.0.json.gz174Loading a file takes 60.50s175Creating data frames takes 25.36s176Filling data types and values takes 6.72s177 178 Processing a file takes 17.47s179Concatenating time series takes 0.42s180Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/25_traj_9078.5.json.gz181Loading a file takes 59.29s182Creating data frames takes 26.95s183Filling data types and values takes 6.76s184 185 Processing a file takes 17.63s186Concatenating time series takes 0.44s187Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/26_traj_9395.5.json.gz188Loading a file takes 57.31s189Creating data frames takes 23.75s190Filling data types and values takes 6.30s191 192 Processing a file takes 16.49s193Concatenating time series takes 0.37s194Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/27_traj_9732.0.json.gz195Loading a file takes 54.97s196Creating data frames takes 24.90s197Filling data types and values takes 6.40s198 199 Processing a file takes 16.96s200Concatenating time series takes 0.37s201Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/28_traj_10097.5.json.gz202Loading a file takes 58.24s203Creating data frames takes 24.12s204Filling data types and values takes 6.43s205 206 Processing a file takes 16.79s207Concatenating time series takes 0.32s208Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/29_traj_10512.0.json.gz209Loading a file takes 59.39s210Creating data frames takes 24.57s211Filling data types and values takes 7.29s212 213 Processing a file takes 17.52s214Concatenating time series takes 0.47s215Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/30_traj_10972.5.json.gz216Loading a file takes 60.42s217Creating data frames takes 24.16s218Filling data types and values takes 6.89s219 220 Processing a file takes 17.00s221Concatenating time series takes 0.34s222Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/31_traj_11505.5.json.gz223Loading a file takes 58.87s224Creating data frames takes 23.21s225Filling data types and values takes 7.17s226 227 Processing a file takes 16.50s228Concatenating time series takes 0.27s229Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/32_traj_12111.5.json.gz230Loading a file takes 65.84s231Creating data frames takes 26.58s232Filling data types and values takes 6.30s233 234 Processing a file takes 15.52s235Concatenating time series takes 0.38s236Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/33_traj_12774.5.json.gz237Loading a file takes 61.71s238Creating data frames takes 25.14s239Filling data types and values takes 6.52s240 241 Processing a file takes 16.32s242Concatenating time series takes 0.39s243Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/34_traj_13491.5.json.gz244Loading a file takes 55.81s245Creating data frames takes 26.12s246Filling data types and values takes 6.60s247 248 Processing a file takes 15.60s249Concatenating time series takes 0.20s250Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/35_traj_14233.0.json.gz251Loading a file takes 57.43s252Creating data frames takes 23.81s253Filling data types and values takes 6.19s254 255 Processing a file takes 14.08s256Concatenating time series takes 0.29s257Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/36_traj_14987.5.json.gz258Loading a file takes 57.86s259Creating data frames takes 24.47s260Filling data types and values takes 6.15s261 262 Processing a file takes 13.55s263Concatenating time series takes 0.24s264Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/37_traj_15755.5.json.gz265Loading a file takes 56.32s266Creating data frames takes 22.76s267Filling data types and values takes 6.05s268 269 Processing a file takes 13.81s270Concatenating time series takes 0.19s271Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/38_traj_16537.0.json.gz272Loading a file takes 55.67s273Creating data frames takes 24.84s274Filling data types and values takes 6.46s275 276 Processing a file takes 14.46s277Concatenating time series takes 0.23s278Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/39_traj_17326.5.json.gz279Loading a file takes 56.81s280Creating data frames takes 20.21s281Filling data types and values takes 5.82s282 283 Processing a file takes 13.41s284Concatenating time series takes 0.20s285Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/40_traj_18122.5.json.gz286Loading a file takes 58.88s287Creating data frames takes 20.32s288Filling data types and values takes 6.00s289 290 Processing a file takes 13.20s291Concatenating time series takes 0.16s292Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/41_traj_18930.0.json.gz293Loading a file takes 59.25s294Creating data frames takes 19.94s295Filling data types and values takes 5.87s296 297 Processing a file takes 13.66s298Concatenating time series takes 0.27s299Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/42_traj_19771.5.json.gz300Loading a file takes 54.57s301Creating data frames takes 19.77s302Filling data types and values takes 5.89s303 304 Processing a file takes 13.24s305Concatenating time series takes 0.21s306Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/43_traj_20660.0.json.gz307Loading a file takes 60.92s308Creating data frames takes 20.48s309Filling data types and values takes 6.00s310 311 Processing a file takes 14.37s312Concatenating time series takes 0.20s313Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/44_traj_21649.0.json.gz314Loading a file takes 57.37s315Creating data frames takes 21.20s316Filling data types and values takes 6.19s317 318 Processing a file takes 13.97s319Concatenating time series takes 0.42s320Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/45_traj_22737.5.json.gz321Loading a file takes 62.23s322Creating data frames takes 21.82s323Filling data types and values takes 6.08s324 325 Processing a file takes 13.22s326Concatenating time series takes 0.25s327Working on file /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/trajectory/46_traj_end.json.gz328Loading a file takes 34.08s329Creating data frames takes 12.56s330Filling data types and values takes 3.20s331 332 Processing a file takes 10.25s333Concatenating time series takes 0.14s334Saving all the section time series takes 169.48s335Saving all the junction time series takes 24.74s336Saving network entrance and exit takes 0.14s337Aggregating the raw statistics to different intervals...338
  0%|          | 0/1525 [00:00<?, ?it/s]
  0%|          | 1/1525 [00:01<27:44,  1.09s/it]
  0%|          | 3/1525 [00:06<57:53,  2.28s/it]
 20%|█▉        | 298/1525 [00:06<00:18, 67.26it/s]
 28%|██▊       | 432/1525 [00:07<00:12, 88.48it/s]
 34%|███▍      | 520/1525 [00:08<00:10, 92.99it/s]
 38%|███▊      | 581/1525 [00:09<00:10, 87.92it/s]
 41%|████      | 624/1525 [00:09<00:10, 89.26it/s]
 43%|████▎     | 656/1525 [00:09<00:09, 94.86it/s]
 45%|████▍     | 683/1525 [00:09<00:08, 102.09it/s]
 46%|████▋     | 707/1525 [00:09<00:07, 109.18it/s]
 48%|████▊     | 729/1525 [00:10<00:07, 113.40it/s]
 49%|████▉     | 748/1525 [00:10<00:06, 121.28it/s]
 50%|█████     | 767/1525 [00:10<00:06, 120.57it/s]
 51%|█████▏    | 785/1525 [00:10<00:05, 128.08it/s]
 53%|█████▎    | 802/1525 [00:10<00:05, 132.27it/s]
 54%|█████▎    | 819/1525 [00:10<00:05, 124.93it/s]
 55%|█████▍    | 834/1525 [00:10<00:05, 124.37it/s]
 56%|█████▌    | 848/1525 [00:11<00:06, 106.56it/s]
 56%|█████▋    | 860/1525 [00:11<00:06, 101.57it/s]
 57%|█████▋    | 871/1525 [00:11<00:06, 96.98it/s] 
 58%|█████▊    | 882/1525 [00:11<00:08, 79.87it/s]
 59%|█████▉    | 898/1525 [00:11<00:06, 95.38it/s]
 60%|█████▉    | 909/1525 [00:11<00:06, 95.86it/s]
 61%|██████    | 929/1525 [00:11<00:05, 107.60it/s]
 62%|██████▏   | 946/1525 [00:12<00:05, 101.89it/s]
 63%|██████▎   | 968/1525 [00:12<00:04, 125.38it/s]
 64%|██████▍   | 983/1525 [00:12<00:04, 129.06it/s]
 65%|██████▌   | 997/1525 [00:12<00:04, 122.15it/s]
 66%|██████▌   | 1010/1525 [00:12<00:04, 115.98it/s]
 67%|██████▋   | 1022/1525 [00:12<00:05, 96.52it/s] 
 68%|██████▊   | 1033/1525 [00:12<00:05, 86.77it/s]
 68%|██████▊   | 1043/1525 [00:13<00:06, 78.58it/s]
 69%|██████▉   | 1052/1525 [00:13<00:05, 80.11it/s]
 70%|███████   | 1068/1525 [00:13<00:04, 95.97it/s]
 71%|███████   | 1079/1525 [00:13<00:04, 90.68it/s]
 72%|███████▏  | 1094/1525 [00:13<00:04, 99.38it/s]
 73%|███████▎  | 1111/1525 [00:13<00:03, 115.88it/s]
 74%|███████▎  | 1124/1525 [00:13<00:04, 94.88it/s] 
 74%|███████▍  | 1135/1525 [00:14<00:04, 89.35it/s]
 76%|███████▌  | 1155/1525 [00:14<00:03, 114.08it/s]
 77%|███████▋  | 1168/1525 [00:14<00:03, 112.80it/s]
 77%|███████▋  | 1181/1525 [00:14<00:03, 98.19it/s] 
 78%|███████▊  | 1192/1525 [00:14<00:03, 97.69it/s]
 79%|███████▉  | 1207/1525 [00:14<00:03, 104.86it/s]
 80%|████████  | 1221/1525 [00:14<00:02, 110.83it/s]
 81%|████████  | 1237/1525 [00:14<00:02, 121.96it/s]
 82%|████████▏ | 1250/1525 [00:15<00:02, 121.08it/s]
 83%|████████▎ | 1263/1525 [00:15<00:02, 117.16it/s]
 84%|████████▍ | 1284/1525 [00:15<00:01, 140.29it/s]
 86%|████████▌ | 1304/1525 [00:15<00:01, 155.95it/s]
 87%|████████▋ | 1320/1525 [00:15<00:01, 151.58it/s]
 88%|████████▊ | 1336/1525 [00:18<00:10, 18.58it/s] 
 89%|████████▉ | 1357/1525 [00:18<00:06, 27.34it/s]
 90%|████████▉ | 1371/1525 [00:18<00:04, 33.66it/s]
 91%|█████████ | 1384/1525 [00:18<00:03, 40.05it/s]
 92%|█████████▏| 1396/1525 [00:18<00:02, 47.65it/s]
 93%|█████████▎| 1416/1525 [00:18<00:01, 66.32it/s]
 94%|█████████▍| 1432/1525 [00:18<00:01, 79.84it/s]
 95%|█████████▍| 1447/1525 [00:18<00:00, 89.90it/s]
 96%|█████████▌| 1462/1525 [00:19<00:00, 91.02it/s]
 97%|█████████▋| 1475/1525 [00:19<00:00, 94.86it/s]
 98%|█████████▊| 1488/1525 [00:19<00:00, 97.84it/s]
 99%|█████████▉| 1516/1525 [00:19<00:00, 137.19it/s]
100%|██████████| 1525/1525 [00:19<00:00, 78.18it/s] 339Constructing time series for each modality ...340Saving aggregated time series to /home/weijiang/Projects/Netsanut/datasets/simbarca/simulation_sessions/session_096/timeseries/agg_timeseries.pkl341Extracting samples from time series ...342number of samples: 20343Packing the samples into numpy arrays ...344Processing completed!